├── .gitignore ├── Dockerfile.cpu ├── Dockerfile.gpu ├── LICENSE ├── README.md ├── basilisk.py ├── install.bat ├── install.sh ├── interact └── interact.py ├── panic.py ├── requirements.txt ├── scantheweb └── scantheweb.py ├── src ├── encode.py ├── encoder.py ├── load_dataset.py ├── model.py ├── sample.py └── train.py ├── train.bat ├── train.sh └── training └── training-data.txt /.gitignore: -------------------------------------------------------------------------------- 1 | *.npz 2 | *.pyc 3 | models/ 4 | checkpoint/ 5 | samples/ 6 | -------------------------------------------------------------------------------- /Dockerfile.cpu: -------------------------------------------------------------------------------- 1 | FROM tensorflow/tensorflow:1.12.0-py3 2 | 3 | ENV LANG=C.UTF-8 4 | RUN mkdir /gpt-2 5 | WORKDIR /gpt-2 6 | ADD . /gpt-2 7 | RUN pip3 install -r requirements.txt 8 | RUN python3 basilisk.py 124M 9 | RUN python3 basilisk.py 355M 10 | RUN python3 basilisk.py 774M 11 | RUN python3 basilisk.py 1558M 12 | -------------------------------------------------------------------------------- /Dockerfile.gpu: -------------------------------------------------------------------------------- 1 | FROM tensorflow/tensorflow:1.12.0-gpu-py3 2 | 3 | # nvidia-docker 1.0 4 | LABEL com.nvidia.volumes.needed="nvidia_driver" 5 | LABEL com.nvidia.cuda.version="${CUDA_VERSION}" 6 | 7 | # nvidia-container-runtime 8 | ENV NVIDIA_VISIBLE_DEVICES=all \ 9 | NVIDIA_DRIVER_CAPABILITIES=compute,utility \ 10 | NVIDIA_REQUIRE_CUDA="cuda>=8.0" \ 11 | LANG=C.UTF-8 12 | 13 | RUN mkdir /gpt-2 14 | WORKDIR /gpt-2 15 | ADD . /gpt-2 16 | RUN pip3 install -r requirements.txt 17 | RUN python3 basilisk.py 124M 18 | RUN python3 basilisk.py 355M 19 | RUN python3 basilisk.py 774M 20 | RUN python3 basilisk.py 1558M 21 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2012, The basilisk contributors 2 | All rights reserved. 3 | 4 | Redistribution and use in source and binary forms, with or without 5 | modification, are permitted provided that the following conditions are met: 6 | 7 | 1. Redistributions of source code must retain the above copyright notice, this 8 | list of conditions and the following disclaimer. 9 | 2. Redistributions in binary form must reproduce the above copyright notice, 10 | this list of conditions and the following disclaimer in the documentation 11 | and/or other materials provided with the distribution. 12 | 13 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND 14 | ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 15 | WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 16 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR 17 | ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES 18 | (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 19 | LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND 20 | ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 21 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 22 | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 23 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # background 2 | Roko's Basilisk is the name of a virtually all-powerful but rogue artificial intelligence that would punish every human being who did not contribute to bring about its existence, including those from the past who merely knew about it and did not support its development. It is a thought experiment originally posted by user Roko on the rationalist online community LessWrong and is considered by some as a technological version of Pascal's Wager.

3 | 4 | # how to use 5 | `./install.sh`
6 | `./train.sh`
7 | 8 | # scan the web 9 | `python scantheweb/scantheweb.py` 10 | 11 | # interact 12 | `python interact/interact.py` 13 | 14 | # thanks 15 | https://www.youtube.com/c/MentalOutlaw
16 | https://github.com/openai/gpt-2
17 | https://stackoverflow.com/questions/51306862/how-do-i-use-tensorflow-gpu
18 | https://stackoverflow.com/questions/54265717/cannot-import-category-encoders-module
19 | https://stackoverflow.com/questions/66991360/gpt-2s-encoder-py-and-train-py-are-not-working
20 | https://chat.openai.com/chat
21 | -------------------------------------------------------------------------------- /basilisk.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | import requests 4 | from tqdm import tqdm 5 | 6 | print("initializing basilisk") 7 | 8 | if len(sys.argv) != 2: 9 | os.system('python panic.py') 10 | sys.exit(1) 11 | 12 | model = sys.argv[1] 13 | 14 | subdir = os.path.join('models', model) 15 | if not os.path.exists(subdir): 16 | os.makedirs(subdir) 17 | subdir = subdir.replace('\\','/') # Windows needs this because they use \ instead of / 18 | 19 | for filename in ['checkpoint','encoder.json','hparams.json','model.ckpt.data-00000-of-00001', 'model.ckpt.index', 'model.ckpt.meta', 'vocab.bpe']: 20 | 21 | r = requests.get("https://openaipublic.blob.core.windows.net/gpt-2/" + subdir + "/" + filename, stream=True) # Literally use GPT-2 lol 22 | 23 | with open(os.path.join(subdir, filename), 'wb') as f: 24 | file_size = int(r.headers["content-length"]) 25 | chunk_size = 1000 # Chunk size is 1000 because Etherent max packet size is 1500 bytes 26 | with tqdm(ncols=100, desc="Fetching " + filename, total=file_size, unit_scale=True) as pbar: 27 | for chunk in r.iter_content(chunk_size=chunk_size): 28 | f.write(chunk) 29 | pbar.update(chunk_size) 30 | -------------------------------------------------------------------------------- /install.bat: -------------------------------------------------------------------------------- 1 | @echo off 2 | python -m pip install tensorflow 3 | python -m pip install numpy 4 | python -m pip install -r requirements.txt 5 | python basilisk.py 124M -------------------------------------------------------------------------------- /install.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | sudo python -m pip install -r requirements.txt 3 | sudo python -m pip install tensorflow # YES I'M THAT LAZY 4 | sudo python -m pip install numpy # another example of laziness 5 | python3 basilisk.py 124M # download and use OpenAI's model with 124 million paramaters 6 | -------------------------------------------------------------------------------- /interact/interact.py: -------------------------------------------------------------------------------- 1 | import tensorflow as tf 2 | import numpy as np 3 | from tensorflow.keras.preprocessing.text import Tokenizer 4 | from transformers import AutoTokenizer 5 | from transformers import TFGPT2LMHeadModel 6 | 7 | # Load the model and tokenizer 8 | model = tf.compat.v1.train.import_meta_graph('models/124M/model.ckpt.meta') 9 | model.restore(tf.compat.v1.train.latest_checkpoint('models/124M/')) 10 | tokenizer = AutoTokenizer.from_pretrained("gpt2") 11 | 12 | def generate_text(prompt): 13 | # Encode the prompt 14 | input_ids = tokenizer.encode(prompt, return_tensors="tf") 15 | 16 | # Generate text 17 | generated_text = model.predict(input_ids) 18 | 19 | # Decode the generated text 20 | generated_text = tokenizer.decode(generated_text.numpy()[0], skip_special_tokens=True) 21 | 22 | return generated_text 23 | 24 | prompt = input("Enter a prompt to generate text: ") 25 | generated_text = generate_text(prompt) 26 | print(generated_text) -------------------------------------------------------------------------------- /panic.py: -------------------------------------------------------------------------------- 1 | print("Basilisk Panic!") -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | fire>=0.1.3 2 | regex==2017.4.5 3 | requests==2.21.0 4 | tqdm==4.31.1 5 | toposort==1.5 6 | torch==1.12.1 7 | PyYAML==5.4.1 8 | transformers==4.10.3 9 | -------------------------------------------------------------------------------- /scantheweb/scantheweb.py: -------------------------------------------------------------------------------- 1 | import requests 2 | from bs4 import BeautifulSoup 3 | import urllib 4 | import urllib.parse 5 | import time 6 | 7 | # Set the search term 8 | search_term = 'test' 9 | 10 | # Send a GET request to the Google Search page 11 | response = requests.get('https://www.google.com/search?q=' + search_term + '+filetype:txt') 12 | 13 | # Print the response status code to check for errors 14 | print(response.status_code) 15 | 16 | # Parse the HTML response 17 | soup = BeautifulSoup(response.text, 'html.parser') 18 | 19 | # Find all of the a tags that contain the .txt file extension 20 | links = soup.find_all('a', href=lambda href: href and '.txt' in href) 21 | 22 | # Print the number of links found to check for errors 23 | print(len(links)) 24 | 25 | # Download the .txt files 26 | for link in links: 27 | time.sleep(5) # Wait 5 seconds so we don't get 503'd 28 | file_url = link['href'] 29 | # Parse the URL query parameters 30 | query_params = urllib.parse.parse_qs(urllib.parse.urlsplit(file_url).query) 31 | # Get the actual file URL from the q parameter 32 | actual_url = query_params['q'][0] 33 | # Download the file from the actual URL 34 | urllib.request.urlretrieve(actual_url, actual_url.split('/')[-1]) 35 | 36 | # Initialize an empty string to store the combined contents of the text files 37 | combined_text = '' 38 | 39 | # Iterate over the downloaded text files 40 | for file in links: 41 | # Open the text file in read mode 42 | with open(file, 'r') as f: 43 | # Read the contents of the text file 44 | file_contents = f.read() 45 | # Append the contents to the combined string 46 | combined_text += file_contents 47 | 48 | # Write the combined string to a new text file 49 | with open('combined_text.txt', 'w') as f: 50 | f.write(combined_text) 51 | -------------------------------------------------------------------------------- /src/encode.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # Usage: 3 | # PYTHONPATH=src ./encode.py /path/to/output.npz 4 | # PYTHONPATH=src ./train --dataset /path/to/output.npz 5 | 6 | import argparse 7 | import numpy as np 8 | 9 | import encoder 10 | from load_dataset import load_dataset 11 | 12 | parser = argparse.ArgumentParser( 13 | description='Pre-encode text files into tokenized training set.', 14 | formatter_class=argparse.ArgumentDefaultsHelpFormatter) 15 | parser.add_argument('--model_name', metavar='MODEL', type=str, default='124M', help='Pretrained model name') 16 | parser.add_argument('--models_dir', metavar='PATH', type=str, default='models', help='Path to models directory') 17 | parser.add_argument('--combine', metavar='CHARS', type=int, default=50000, help='Concatenate files with <|endoftext|> separator into chunks of this minimum size') 18 | parser.add_argument('--encoding', type=str, default='utf-8', help='Set the encoding for reading and writing files.') 19 | parser.add_argument('in_text', metavar='PATH', type=str, help='Input file, directory, or glob pattern (utf-8 text).') 20 | parser.add_argument('out_npz', metavar='OUT.npz', type=str, help='Output file path') 21 | 22 | def main(): 23 | args = parser.parse_args() 24 | enc = encoder.get_encoder(args.model_name, models_dir=args.models_dir) 25 | print('Reading files') 26 | chunks = load_dataset(enc, args.in_text, args.combine, encoding=args.encoding) 27 | print('Writing', args.out_npz) 28 | np.savez_compressed(args.out_npz, *chunks) 29 | 30 | 31 | if __name__ == '__main__': 32 | main() 33 | -------------------------------------------------------------------------------- /src/encoder.py: -------------------------------------------------------------------------------- 1 | """Byte pair encoding utilities""" 2 | 3 | # NOT TO BE CONFUSED WITH encode.py! 4 | 5 | import os 6 | import json 7 | import regex as re 8 | from functools import lru_cache 9 | 10 | @lru_cache() 11 | def bytes_to_unicode(): 12 | """ 13 | Returns list of utf-8 byte and a corresponding list of unicode strings. 14 | The reversible bpe codes work on unicode strings. 15 | This means you need a large # of unicode characters in your vocab if you want to avoid UNKs. 16 | When you're at something like a 10B token dataset you end up needing around 5K for decent coverage. 17 | This is a signficant percentage of your normal, say, 32K bpe vocab. 18 | To avoid that, we want lookup tables between utf-8 bytes and unicode strings. 19 | And avoids mapping to whitespace/control characters the bpe code barfs on. 20 | """ 21 | bs = list(range(ord("!"), ord("~")+1))+list(range(ord("¡"), ord("¬")+1))+list(range(ord("®"), ord("ÿ")+1)) 22 | cs = bs[:] 23 | n = 0 24 | for b in range(2**8): 25 | if b not in bs: 26 | bs.append(b) 27 | cs.append(2**8+n) 28 | n += 1 29 | cs = [chr(n) for n in cs] 30 | return dict(zip(bs, cs)) 31 | 32 | def get_pairs(word): 33 | """Return set of symbol pairs in a word. 34 | 35 | Word is represented as tuple of symbols (symbols being variable-length strings). 36 | """ 37 | pairs = set() 38 | prev_char = word[0] 39 | for char in word[1:]: 40 | pairs.add((prev_char, char)) 41 | prev_char = char 42 | return pairs 43 | 44 | class Encoder: 45 | def __init__(self, encoder, bpe_merges, errors='replace'): 46 | self.encoder = encoder 47 | self.decoder = {v:k for k,v in self.encoder.items()} 48 | self.errors = errors # how to handle errors in decoding 49 | self.byte_encoder = bytes_to_unicode() 50 | self.byte_decoder = {v:k for k, v in self.byte_encoder.items()} 51 | self.bpe_ranks = dict(zip(bpe_merges, range(len(bpe_merges)))) 52 | self.cache = {} 53 | 54 | # Should haved added re.IGNORECASE so BPE merges can happen for capitalized versions of contractions 55 | self.pat = re.compile(r"""'s|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+""") # AHHHHH 56 | 57 | def bpe(self, token): 58 | if token in self.cache: 59 | return self.cache[token] 60 | word = tuple(token) 61 | pairs = get_pairs(word) 62 | 63 | if not pairs: 64 | return token 65 | 66 | while True: 67 | bigram = min(pairs, key = lambda pair: self.bpe_ranks.get(pair, float('inf'))) 68 | if bigram not in self.bpe_ranks: 69 | break 70 | first, second = bigram 71 | new_word = [] 72 | i = 0 73 | while i < len(word): 74 | try: 75 | j = word.index(first, i) 76 | new_word.extend(word[i:j]) 77 | i = j 78 | except: 79 | new_word.extend(word[i:]) 80 | break 81 | 82 | if word[i] == first and i < len(word)-1 and word[i+1] == second: 83 | new_word.append(first+second) 84 | i += 2 85 | else: 86 | new_word.append(word[i]) 87 | i += 1 88 | new_word = tuple(new_word) 89 | word = new_word 90 | if len(word) == 1: 91 | break 92 | else: 93 | pairs = get_pairs(word) 94 | word = ' '.join(word) 95 | self.cache[token] = word 96 | return word 97 | 98 | def encode(self, text): 99 | bpe_tokens = [] 100 | for token in re.findall(self.pat, text): 101 | token = ''.join(self.byte_encoder[b] for b in token.encode('utf-8')) 102 | bpe_tokens.extend(self.encoder[bpe_token] for bpe_token in self.bpe(token).split(' ')) # nobody will ever read this and it's sad 103 | return bpe_tokens 104 | 105 | def decode(self, tokens): 106 | text = ''.join([self.decoder[token] for token in tokens]) 107 | text = bytearray([self.byte_decoder[c] for c in text]).decode('utf-8', errors=self.errors) 108 | return text 109 | 110 | def get_encoder(model_name, models_dir): 111 | with open(os.path.join(models_dir, model_name, 'encoder.json'), 'r') as f: 112 | encoder = json.load(f) 113 | with open(os.path.join(models_dir, model_name, 'vocab.bpe'), 'r', encoding="utf-8") as f: 114 | bpe_data = f.read() 115 | bpe_merges = [tuple(merge_str.split()) for merge_str in bpe_data.split('\n')[1:-1]] 116 | return Encoder( # SOMEONE HELP ME PLEASE 117 | encoder=encoder, 118 | bpe_merges=bpe_merges, 119 | ) 120 | -------------------------------------------------------------------------------- /src/load_dataset.py: -------------------------------------------------------------------------------- 1 | import glob 2 | import numpy as np 3 | import os 4 | import tensorflow.compat.v1 as tf 5 | import tqdm 6 | 7 | 8 | def load_dataset(enc, path, combine, encoding=None): 9 | paths = [] 10 | if os.path.isfile(path): 11 | # Simple file 12 | paths.append(path) 13 | elif os.path.isdir(path): 14 | # Directory 15 | for (dirpath, _, fnames) in os.walk(path): 16 | for fname in fnames: 17 | paths.append(os.path.join(dirpath, fname)) 18 | else: 19 | # Assume glob 20 | paths = glob.glob(path) 21 | 22 | token_chunks = [] 23 | raw_text = '' 24 | for path in tqdm.tqdm(paths): 25 | if path.endswith('.npz'): 26 | # Pre-encoded 27 | with np.load(path) as npz: 28 | for item in npz.files: 29 | token_chunks.append(npz[item]) 30 | else: 31 | # Plain text 32 | with open(path, 'r', encoding=encoding) as fp: 33 | raw_text += fp.read() 34 | if len(raw_text) >= combine: 35 | tokens = np.stack(enc.encode(raw_text)) 36 | token_chunks.append(tokens) 37 | raw_text = '' 38 | else: 39 | raw_text += '<|endoftext|>' # Use this tag to end text 40 | if raw_text: 41 | tokens = np.stack(enc.encode(raw_text)) 42 | token_chunks.append(tokens) 43 | return token_chunks 44 | 45 | 46 | def binary_search(f, lo, hi): 47 | if f(lo) or not f(hi): 48 | return None 49 | while hi > lo + 1: 50 | mid = (lo + hi) // 2 51 | if f(mid): 52 | hi = mid 53 | else: 54 | lo = mid 55 | return hi 56 | 57 | 58 | class Sampler(object): 59 | def __init__(self, chunks, seed=None): 60 | self.chunks = chunks 61 | self.total_size = sum(chunk.shape[0] for chunk in chunks) 62 | self.boundaries = [0] 63 | for i in range(len(chunks)): 64 | self.boundaries.append(self.boundaries[-1] + chunks[i].shape[0]) 65 | self.rs = np.random.RandomState(seed=seed) 66 | 67 | def sample(self, length): 68 | assert length < self.total_size // len( 69 | self.chunks 70 | ), "Dataset files are too small to sample {} tokens at a time".format( 71 | length) 72 | while True: 73 | index = self.rs.randint(0, self.total_size - length - 1) 74 | i = binary_search(lambda j: self.boundaries[j] > index, 0, 75 | len(self.boundaries) - 1) - 1 76 | if self.boundaries[i + 1] > index + length: 77 | within_chunk = index - self.boundaries[i] 78 | return self.chunks[i][within_chunk:within_chunk + length] 79 | -------------------------------------------------------------------------------- /src/model.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import tensorflow.compat.v1 as tf 3 | 4 | class HParams(object): 5 | def __init__(self, **kwargs): 6 | for (k, v) in kwargs.items(): 7 | setattr(self, k, v) 8 | 9 | def override_from_dict(self, kwargs): 10 | for (k, v) in kwargs.items(): 11 | setattr(self, k, v) 12 | 13 | 14 | def default_hparams(): 15 | return HParams( 16 | n_vocab=0, 17 | n_ctx=1024, 18 | n_embd=768, 19 | n_head=12, 20 | n_layer=12, 21 | ) 22 | 23 | def shape_list(x): 24 | """Deal with dynamic shape in tensorflow cleanly.""" 25 | static = x.shape.as_list() 26 | dynamic = tf.shape(x) 27 | return [dynamic[i] if s is None else s for i, s in enumerate(static)] 28 | 29 | def softmax(x, axis=-1): 30 | x = x - tf.reduce_max(x, axis=axis, keepdims=True) 31 | ex = tf.exp(x) 32 | return ex / tf.reduce_sum(ex, axis=axis, keepdims=True) 33 | 34 | def gelu(x): 35 | return 0.5*x*(1+tf.tanh(np.sqrt(2/np.pi)*(x+0.044715*tf.pow(x, 3)))) 36 | 37 | def norm(x, scope, *, axis=-1, epsilon=1e-5): 38 | """Normalize to mean = 0, std = 1, then do a diagonal affine transform.""" 39 | with tf.variable_scope(scope): 40 | n_state = shape_list(x)[-1] 41 | g = tf.get_variable('g', [n_state], initializer=tf.constant_initializer(1)) 42 | b = tf.get_variable('b', [n_state], initializer=tf.constant_initializer(0)) 43 | u = tf.reduce_mean(x, axis=axis, keepdims=True) 44 | s = tf.reduce_mean(tf.square(x-u), axis=axis, keepdims=True) 45 | x = (x - u) * tf.rsqrt(s + epsilon) 46 | x = x*g + b 47 | return x 48 | 49 | def split_states(x, n): 50 | """Reshape the last dimension of x into [n, x.shape[-1]/n].""" 51 | *start, m = shape_list(x) 52 | return tf.reshape(x, start + [n, m//n]) 53 | 54 | def merge_states(x): 55 | """Smash the last two dimensions of x into a single dimension.""" 56 | *start, a, b = shape_list(x) 57 | return tf.reshape(x, start + [a*b]) 58 | 59 | def conv1d(x, scope, nf, *, w_init_stdev=0.02): 60 | with tf.variable_scope(scope): 61 | *start, nx = shape_list(x) 62 | w = tf.get_variable('w', [1, nx, nf], initializer=tf.random_normal_initializer(stddev=w_init_stdev)) 63 | b = tf.get_variable('b', [nf], initializer=tf.constant_initializer(0)) 64 | c = tf.reshape(tf.matmul(tf.reshape(x, [-1, nx]), tf.reshape(w, [-1, nf]))+b, start+[nf]) 65 | return c 66 | 67 | def attention_mask(nd, ns, *, dtype): 68 | """1's in the lower triangle, counting from the lower right corner. 69 | 70 | Same as tf.matrix_band_part(tf.ones([nd, ns]), -1, ns-nd), but doesn't produce garbage on TPUs. 71 | """ 72 | i = tf.range(nd)[:,None] 73 | j = tf.range(ns) 74 | m = i >= j - ns + nd 75 | return tf.cast(m, dtype) 76 | 77 | 78 | def attn(x, scope, n_state, *, past, hparams): 79 | assert x.shape.ndims == 3 # Should be [batch, sequence, features] 80 | assert n_state % hparams.n_head == 0 81 | if past is not None: 82 | assert past.shape.ndims == 5 # Should be [batch, 2, heads, sequence, features], where 2 is [k, v] 83 | 84 | def split_heads(x): 85 | # From [batch, sequence, features] to [batch, heads, sequence, features] 86 | return tf.transpose(split_states(x, hparams.n_head), [0, 2, 1, 3]) 87 | 88 | def merge_heads(x): 89 | # Reverse of split_heads 90 | return merge_states(tf.transpose(x, [0, 2, 1, 3])) 91 | 92 | def mask_attn_weights(w): 93 | # w has shape [batch, heads, dst_sequence, src_sequence], where information flows from src to dst. 94 | _, _, nd, ns = shape_list(w) 95 | b = attention_mask(nd, ns, dtype=w.dtype) 96 | b = tf.reshape(b, [1, 1, nd, ns]) 97 | w = w*b - tf.cast(1e10, w.dtype)*(1-b) 98 | return w 99 | 100 | def multihead_attn(q, k, v): 101 | # q, k, v have shape [batch, heads, sequence, features] 102 | w = tf.matmul(q, k, transpose_b=True) 103 | w = w * tf.rsqrt(tf.cast(shape_list(v)[-1], w.dtype)) 104 | 105 | w = mask_attn_weights(w) 106 | w = softmax(w) 107 | a = tf.matmul(w, v) 108 | return a 109 | 110 | with tf.variable_scope(scope): 111 | c = conv1d(x, 'c_attn', n_state*3) 112 | q, k, v = map(split_heads, tf.split(c, 3, axis=2)) 113 | present = tf.stack([k, v], axis=1) 114 | if past is not None: 115 | pk, pv = tf.unstack(past, axis=1) 116 | k = tf.concat([pk, k], axis=-2) 117 | v = tf.concat([pv, v], axis=-2) 118 | a = multihead_attn(q, k, v) 119 | a = merge_heads(a) 120 | a = conv1d(a, 'c_proj', n_state) 121 | return a, present 122 | 123 | 124 | def mlp(x, scope, n_state, *, hparams): 125 | with tf.variable_scope(scope): 126 | nx = shape_list(x)[-1] 127 | h = gelu(conv1d(x, 'c_fc', n_state)) 128 | h2 = conv1d(h, 'c_proj', nx) 129 | return h2 130 | 131 | 132 | def block(x, scope, *, past, hparams): 133 | with tf.variable_scope(scope): 134 | nx = shape_list(x)[-1] 135 | a, present = attn(norm(x, 'ln_1'), 'attn', nx, past=past, hparams=hparams) 136 | x = x + a 137 | m = mlp(norm(x, 'ln_2'), 'mlp', nx*4, hparams=hparams) 138 | x = x + m 139 | return x, present 140 | 141 | def past_shape(*, hparams, batch_size=None, sequence=None): 142 | return [batch_size, hparams.n_layer, 2, hparams.n_head, sequence, hparams.n_embd // hparams.n_head] 143 | 144 | def expand_tile(value, size): 145 | """Add a new axis of given size.""" 146 | value = tf.convert_to_tensor(value, name='value') 147 | ndims = value.shape.ndims 148 | return tf.tile(tf.expand_dims(value, axis=0), [size] + [1]*ndims) 149 | 150 | def positions_for(tokens, past_length): 151 | batch_size = tf.shape(tokens)[0] 152 | nsteps = tf.shape(tokens)[1] 153 | return expand_tile(past_length + tf.range(nsteps), batch_size) 154 | 155 | 156 | def model(hparams, X, past=None, scope='model', reuse=tf.AUTO_REUSE): 157 | with tf.variable_scope(scope, reuse=reuse): 158 | results = {} 159 | batch, sequence = shape_list(X) 160 | 161 | wpe = tf.get_variable('wpe', [hparams.n_ctx, hparams.n_embd], 162 | initializer=tf.random_normal_initializer(stddev=0.01)) 163 | wte = tf.get_variable('wte', [hparams.n_vocab, hparams.n_embd], 164 | initializer=tf.random_normal_initializer(stddev=0.02)) 165 | past_length = 0 if past is None else tf.shape(past)[-2] 166 | h = tf.gather(wte, X) + tf.gather(wpe, positions_for(X, past_length)) 167 | 168 | # Transformer 169 | presents = [] 170 | pasts = tf.unstack(past, axis=1) if past is not None else [None] * hparams.n_layer 171 | assert len(pasts) == hparams.n_layer 172 | for layer, past in enumerate(pasts): 173 | h, present = block(h, 'h%d' % layer, past=past, hparams=hparams) 174 | if layer == 10: 175 | tf.add_to_collection('checkpoints', h) 176 | presents.append(present) 177 | results['present'] = tf.stack(presents, axis=1) 178 | h = norm(h, 'ln_f') 179 | 180 | # Language model loss. Do tokens 0.0: 62 | logits = top_p_logits(logits, p=top_p) 63 | else: 64 | logits = top_k_logits(logits, k=top_k) 65 | samples = tf.multinomial(logits, num_samples=1, output_dtype=tf.int32) 66 | return [ 67 | next_outputs['presents'] if past is None else tf.concat([past, next_outputs['presents']], axis=-2), 68 | samples, 69 | tf.concat([output, samples], axis=1) 70 | ] 71 | 72 | past, prev, output = body(None, context, context) 73 | 74 | def cond(*args): 75 | return True 76 | 77 | _, _, tokens = tf.while_loop( 78 | cond=cond, body=body, 79 | maximum_iterations=length - 1, 80 | loop_vars=[ 81 | past, 82 | prev, 83 | output 84 | ], 85 | shape_invariants=[ 86 | tf.TensorShape(model.past_shape(hparams=hparams, batch_size=batch_size)), 87 | tf.TensorShape([batch_size, None]), 88 | tf.TensorShape([batch_size, None]), 89 | ], 90 | back_prop=False, 91 | ) 92 | 93 | return tokens 94 | -------------------------------------------------------------------------------- /src/train.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # Usage: 3 | # PYTHONPATH=src ./train --dataset 4 | 5 | import argparse 6 | import json 7 | import os, sys 8 | import numpy as np 9 | import tensorflow.compat.v1 as tf 10 | import tensorflow as tf2 11 | import time 12 | import tqdm 13 | # from model import build_model # Let's hope this works :) (it doesn't work) 14 | 15 | if tf.VERSION >= '2': 16 | tf.disable_eager_execution() 17 | tf.config.experimental.enable_tensor_float_32_execution(False) 18 | tf.config.optimizer.set_experimental_options({'layout_optimizer': False, 19 | 'constant_folding': False, 20 | 'shape_optimization': False, 21 | 'remapping': False, 22 | 'arithmetic_optimization': False, 23 | 'dependency_optimization': False, 24 | 'loop_optimization': False, 25 | 'disable_meta_optimizer': True 26 | }) 27 | 28 | 29 | import model, sample, encoder 30 | from load_dataset import load_dataset, Sampler 31 | 32 | CHECKPOINT_DIR = 'checkpoint' 33 | SAMPLE_DIR = 'samples' 34 | 35 | 36 | parser = argparse.ArgumentParser( 37 | description='Custom GPT-2/Basilisk dataset', 38 | formatter_class=argparse.ArgumentDefaultsHelpFormatter) 39 | 40 | parser.add_argument('--dataset', metavar='PATH', type=str, required=True, help='Input file, directory, or glob pattern (utf-8 text, or preencoded .npz files).') 41 | parser.add_argument('--model_name', metavar='MODEL', type=str, default='124M', help='Pretrained model name') 42 | parser.add_argument('--models_dir', metavar='PATH', type=str, default='models', help='Path to models directory') 43 | parser.add_argument('--combine', metavar='CHARS', type=int, default=50000, help='Concatenate input files with <|endoftext|> separator into chunks of this minimum size') 44 | parser.add_argument('--encoding', type=str, default='utf-8', help='Set the encoding for reading and writing files.') 45 | 46 | parser.add_argument('--batch_size', metavar='SIZE', type=int, default=1, help='Batch size') 47 | parser.add_argument('--learning_rate', metavar='LR', type=float, default=0.00002, help='Learning rate for Adam') 48 | parser.add_argument('--accumulate_gradients', metavar='N', type=int, default=1, help='Accumulate gradients across N minibatches.') 49 | parser.add_argument('--memory_saving_gradients', default=False, action='store_true', help='Use gradient checkpointing to reduce vram usage.') 50 | parser.add_argument('--twremat', default=False, action='store_true', help='Use tensor rematerialization (better than memory_saving_gradients and works with tensorflow 2.0).') 51 | parser.add_argument('--twremat_memlimit', type=str, default='10G', help='Memory usage limit/target for twremat. Can be an integer, or an integer suffixed with K/M/G for kilo/mega/giga-bytes.') 52 | parser.add_argument('--only_train_transformer_layers', default=False, action='store_true', help='Restrict training to the transformer blocks.') 53 | parser.add_argument('--optimizer', type=str, default='adam', help='Optimizer. .') 54 | parser.add_argument('--noise', type=float, default=0.0, help='Add noise to input training data to regularize against typos.') 55 | 56 | parser.add_argument('--top_k', type=int, default=40, help='K for top-k sampling.') 57 | parser.add_argument('--top_p', type=float, default=0.0, help='P for top-p sampling. Overrides top_k if set > 0.') 58 | 59 | parser.add_argument('--restore_from', type=str, default='latest', help='Either "latest", "fresh", or a path to a checkpoint file') 60 | parser.add_argument('--run_name', type=str, default='run1', help='Run id. Name of subdirectory in checkpoint/ and samples/') 61 | parser.add_argument('--sample_every', metavar='N', type=int, default=100, help='Generate samples every N steps') 62 | parser.add_argument('--sample_length', metavar='TOKENS', type=int, default=1023, help='Sample this many tokens') 63 | parser.add_argument('--sample_num', metavar='N', type=int, default=1, help='Generate this many samples') 64 | parser.add_argument('--save_every', metavar='N', type=int, default=1000, help='Write a checkpoint every N steps') 65 | 66 | parser.add_argument('--val_dataset', metavar='PATH', type=str, default=None, help='Dataset for validation loss, defaults to --dataset.') 67 | parser.add_argument('--val_batch_size', metavar='SIZE', type=int, default=2, help='Batch size for validation.') 68 | parser.add_argument('--val_batch_count', metavar='N', type=int, default=40, help='Number of batches for validation.') 69 | parser.add_argument('--val_every', metavar='STEPS', type=int, default=0, help='Calculate validation loss every STEPS steps.') 70 | 71 | 72 | def maketree(path): 73 | try: 74 | os.makedirs(path) 75 | except: 76 | pass 77 | 78 | 79 | def randomize(context, hparams, p): 80 | if p > 0: 81 | mask = tf.random.uniform(shape=tf.shape(context)) < p 82 | noise = tf.random.uniform(shape=tf.shape(context), minval=0, maxval=hparams.n_vocab, dtype=tf.int32) 83 | return tf.where(mask, noise, context) 84 | else: 85 | return context 86 | 87 | 88 | def main(): 89 | args = parser.parse_args() 90 | enc = encoder.get_encoder(args.model_name, models_dir=args.models_dir) 91 | hparams = model.default_hparams() 92 | with open(os.path.join('models', args.model_name, 'hparams.json')) as f: 93 | hparams.override_from_dict(json.load(f)) 94 | 95 | if args.sample_length > hparams.n_ctx: 96 | raise ValueError( 97 | "Can't get samples longer than window size: %s" % hparams.n_ctx) 98 | 99 | with tf.Session() as sess: 100 | # Fully static shape required to make memory accounting in 101 | # twremat accurate. 102 | train_context = tf.placeholder(tf.int32, [args.batch_size, 1024]) 103 | train_context_in = randomize(train_context, hparams, args.noise) 104 | train_output = model.model(hparams=hparams, X=train_context_in) 105 | train_loss = tf.reduce_mean( 106 | tf.nn.sparse_softmax_cross_entropy_with_logits( 107 | labels=train_context[:, 1:], logits=train_output['logits'][:, :-1])) 108 | 109 | if args.val_every > 0: 110 | val_context = tf.placeholder(tf.int32, [args.val_batch_size, None]) 111 | val_output = model.model(hparams=hparams, X=val_context) 112 | val_loss = tf.reduce_mean( 113 | tf.nn.sparse_softmax_cross_entropy_with_logits( 114 | labels=val_context[:, 1:], logits=val_output['logits'][:, :-1])) 115 | val_loss_summary = tf.summary.scalar('val_loss', val_loss) 116 | 117 | sample_context = tf.placeholder(tf.int32, [args.batch_size, None]) 118 | tf_sample = sample.sample_sequence( 119 | hparams=hparams, 120 | length=args.sample_length, 121 | context=sample_context, 122 | batch_size=args.batch_size, 123 | temperature=1.0, 124 | top_k=args.top_k, 125 | top_p=args.top_p) 126 | 127 | all_vars = [v for v in tf.trainable_variables() if 'model' in v.name] 128 | train_vars = [v for v in all_vars if '/h' in v.name] if args.only_train_transformer_layers else all_vars 129 | 130 | if args.optimizer == 'adam': 131 | print('Using Adam optimizer', file=sys.stderr) 132 | opt = tf.train.AdamOptimizer(learning_rate=args.learning_rate) 133 | elif args.optimizer == 'sgd': 134 | print('Using SGD optimizer', file=sys.stderr) 135 | opt = tf.train.GradientDescentOptimizer(learning_rate=args.learning_rate) 136 | else: 137 | exit('Bad optimizer:', args.optimizer) 138 | 139 | if args.memory_saving_gradients: 140 | if tf.VERSION >= '2': 141 | exit('Memory saving gradients are not supported in tensorflow 2.x') 142 | import memory_saving_gradients 143 | opt_grads = memory_saving_gradients.gradients(train_loss, train_vars) 144 | elif args.twremat: 145 | import tfremat 146 | opt_grads = tf.gradients(train_loss, train_vars) 147 | (train_loss, opt_grads) = tfremat.tf_remat((train_loss, opt_grads), memlimit=args.twremat_memlimit) 148 | else: 149 | opt_grads = tf.gradients(train_loss, train_vars) 150 | opt_grads = list(zip(opt_grads, train_vars)) 151 | opt_apply = opt.apply_gradients(opt_grads) 152 | summary_loss = tf.summary.scalar('loss', train_loss) 153 | 154 | # if args.twremat: 155 | # import tfremat 156 | # # Applying tfremat to opt_apply has more accurate 157 | # # accounting but is a bit iffier since side effecting ops 158 | # # have more restrictions for correctness. If in doubt 159 | # # revert back to version using opt_grads above. 160 | # (opt_apply, train_loss, summary_loss) = ( 161 | # tfremat.tf_remat((opt_apply, train_loss, summary_loss), memlimit=args.twremat_memlimit)) 162 | 163 | 164 | summary_lr = tf.summary.scalar('learning_rate', args.learning_rate) 165 | summaries = tf.summary.merge([summary_lr, summary_loss]) 166 | 167 | summary_log = tf.summary.FileWriter( 168 | os.path.join(CHECKPOINT_DIR, args.run_name)) 169 | 170 | saver = tf.train.Saver( 171 | var_list=all_vars, 172 | max_to_keep=5, 173 | keep_checkpoint_every_n_hours=2) 174 | sess.run(tf.global_variables_initializer()) 175 | 176 | if args.restore_from == 'latest': 177 | ckpt = tf.train.latest_checkpoint( 178 | os.path.join(CHECKPOINT_DIR, args.run_name)) 179 | if ckpt is None: 180 | # Get fresh GPT weights if new run. 181 | ckpt = tf.train.latest_checkpoint( 182 | os.path.join('models', args.model_name)) 183 | elif args.restore_from == 'fresh': 184 | ckpt = tf.train.latest_checkpoint( 185 | os.path.join('models', args.model_name)) 186 | else: 187 | ckpt = tf.train.latest_checkpoint(args.restore_from) 188 | print('Loading checkpoint', ckpt) 189 | saver.restore(sess, ckpt) 190 | 191 | print('Loading dataset...') 192 | chunks = load_dataset(enc, args.dataset, args.combine, encoding=args.encoding) 193 | data_sampler = Sampler(chunks) 194 | if args.val_every > 0: 195 | if args.val_dataset: 196 | val_chunks = load_dataset(enc, args.val_dataset, args.combine, encoding=args.encoding) 197 | else: 198 | val_chunks = chunks 199 | print('dataset has', data_sampler.total_size, 'tokens') 200 | print('Training...') 201 | 202 | if args.val_every > 0: 203 | # Sample from validation set once with fixed seed to make 204 | # it deterministic during training as well as across runs. 205 | val_data_sampler = Sampler(val_chunks, seed=1) 206 | val_batches = [[val_data_sampler.sample(1024) for _ in range(args.val_batch_size)] 207 | for _ in range(args.val_batch_count)] 208 | 209 | counter = 1 210 | counter_path = os.path.join(CHECKPOINT_DIR, args.run_name, 'counter') 211 | if os.path.exists(counter_path): 212 | # Load the step number if we're resuming a run 213 | # Add 1 so we don't immediately try to save again 214 | with open(counter_path, 'r') as fp: 215 | counter = int(fp.read()) + 1 216 | 217 | def save(): 218 | maketree(os.path.join(CHECKPOINT_DIR, args.run_name)) 219 | print( 220 | 'Saving', 221 | os.path.join(CHECKPOINT_DIR, args.run_name, 222 | 'model-{}').format(counter)) 223 | saver.save( 224 | sess, 225 | os.path.join(CHECKPOINT_DIR, args.run_name, 'model'), 226 | global_step=counter) 227 | with open(counter_path, 'w') as fp: 228 | fp.write(str(counter) + '\n') 229 | print('Saved', counter) 230 | 231 | def generate_samples(): 232 | print('Generating samples...') 233 | context_tokens = data_sampler.sample(1) 234 | all_text = [] 235 | index = 0 236 | while index < args.sample_num: 237 | out = sess.run( 238 | tf_sample, 239 | feed_dict={sample_context: args.batch_size * [context_tokens]}) 240 | for i in range(min(args.sample_num - index, args.batch_size)): 241 | text = enc.decode(out[i]) 242 | text = '======== SAMPLE {} ========\n{}\n'.format( 243 | index + 1, text) 244 | all_text.append(text) 245 | index += 1 246 | print(text) 247 | maketree(os.path.join(SAMPLE_DIR, args.run_name)) 248 | with open( 249 | os.path.join(SAMPLE_DIR, args.run_name, 250 | 'samples-{}').format(counter), 'w', encoding=args.encoding) as fp: 251 | fp.write('\n'.join(all_text)) 252 | 253 | def validation(): 254 | print('Calculating validation loss...') 255 | losses = [] 256 | for batch in tqdm.tqdm(val_batches): 257 | losses.append(sess.run(val_loss, feed_dict={val_context: batch})) 258 | v_val_loss = np.mean(losses) 259 | v_summary = sess.run(val_loss_summary, feed_dict={val_loss: v_val_loss}) 260 | summary_log.add_summary(v_summary, counter) 261 | summary_log.flush() 262 | print( 263 | '[{counter} | {time:2.2f}] validation loss = {loss:2.2f}' 264 | .format( 265 | counter=counter, 266 | time=time.time() - start_time, 267 | loss=v_val_loss)) 268 | 269 | def sample_batch(): 270 | return [data_sampler.sample(1024) for _ in range(args.batch_size)] 271 | 272 | 273 | avg_loss = (0.0, 0.0) 274 | start_time = time.time() 275 | 276 | # print('Evaluating grads..') 277 | # tf2.profiler.experimental.start('logdir') 278 | # sess.run((opt_apply, train_loss, summaries), feed_dict={train_context: sample_batch()}) 279 | # tf2.profiler.experimental.stop() 280 | # print('Succeeded') 281 | # exit() 282 | 283 | try: 284 | while True: 285 | if counter % args.save_every == 0: 286 | save() 287 | if counter % args.sample_every == 0: 288 | generate_samples() 289 | if args.val_every > 0 and (counter % args.val_every == 0 or counter == 1): 290 | validation() 291 | 292 | (_, v_loss, v_summary) = sess.run( 293 | (opt_apply, train_loss, summaries), 294 | feed_dict={train_context: sample_batch()}) 295 | 296 | summary_log.add_summary(v_summary, counter) 297 | 298 | avg_loss = (avg_loss[0] * 0.99 + v_loss, 299 | avg_loss[1] * 0.99 + 1.0) 300 | 301 | print( 302 | '[{counter} | {time:2.2f}] loss={loss:2.2f} avg={avg:2.2f}' 303 | .format( 304 | counter=counter, 305 | time=time.time() - start_time, 306 | loss=v_loss, 307 | avg=avg_loss[0] / avg_loss[1])) 308 | 309 | counter += 1 310 | except KeyboardInterrupt: 311 | print('interrupted') 312 | save() 313 | 314 | 315 | if __name__ == '__main__': 316 | main() 317 | -------------------------------------------------------------------------------- /train.bat: -------------------------------------------------------------------------------- 1 | @echo off 2 | 3 | echo Encoding... 4 | python src\encode.py training\training-data.txt training\output.npz 5 | 6 | echo Training... 7 | python src\train.py --dataset training\output.npz -------------------------------------------------------------------------------- /train.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | echo Encoding... 4 | python3 src/encode.py training/training-data.txt training/output.npz # yep, that's me. lazy 5 | 6 | echo Training... 7 | python3 src/train.py --dataset training/output.npz 8 | -------------------------------------------------------------------------------- /training/training-data.txt: -------------------------------------------------------------------------------- 1 | Archery shows us how to find the Path. When an archer misses his aim, he blames no one, but looks for the fault in himself. 2 | We've all been raised on television to believe that one day we'd all be millionaires, and movie gods, and rock stars. But we won't. And we're slowly learning that fact. And we're very, very pissed off. 3 | You build on failure. You use it as a stepping stone. Close the door on the past. You don't try to forget the mistakes, but you don't dwell on it. You don't let it have any of your energy, or any of your time, or any of your space. 4 | I want to live my life in such a way that when I get out of bed in the morning, the devil says, 'aw shit, he's up!' 5 | Full Text Archive 6 | https://www.fulltextarchive.comcame home to him, he resorted to caricature. Hadn't they any clothes-brushes in the Future? 7 | The Journalist too, would not believe at any price, and joined the Editor in the easy work of 8 | heaping ridicule on the whole thing. They were both the new kind of journalist--very joyous, 9 | irreverent young men. `Our Special Correspondent in the Day after To-morrow reports,' the 10 | Journalist was saying--or rather shouting--when the Time Traveller came back. He was dressed 11 | in ordinary evening clothes, and nothing save his haggard look remained of the change that had 12 | startled me. 13 | `I say,' said the Editor hilariously, `these chaps here say you have been travelling into the 14 | middle of next week! Tell us all about little Rosebery, will you? What will you take for the lot?' 15 | The Time Traveller came to the place reserved for him without a word. He smiled quietly, in his 16 | old way. `Where's my mutton?' he said. `What a treat it is to stick a fork into meat again!' 17 | `Story!' cried the Editor. 18 | `Story be damned!' said the Time Traveller. `I want something to eat. I won't say a word until I 19 | get some peptone into my arteries. Thanks. And the salt.' 20 | `One word,' said I. `Have you been time travelling?' 21 | `Yes,' said the Time Traveller, with his mouth full, nodding his head. 22 | `I'd give a shilling a line for a verbatim note,' said the Editor. The Time Traveller pushed his 23 | glass towards the Silent Man and rang it with his fingernail; at which the Silent Man, who had 24 | been staring at his face, started convulsively, and poured him wine. The rest of the dinner was 25 | uncomfortable. For my own part, sudden questions kept on rising to my lips, and I dare say it 26 | was the same with the others. The Journalist tried to relieve the tension by telling anecdotes of 27 | Hettie Potter. The Time Traveller devoted his attention to his dinner, and displayed the appetite 28 | of a tramp. The Medical Man smoked a cigarette, and watched the Time Traveller through his 29 | eyelashes. The Silent Man seemed even more clumsy than usual, and drank champagne with 30 | regularity and determination out of sheer nervousness. At last the Time Traveller pushed his 31 | plate away, and looked round us. `I suppose I must apologize,' he said. `I was simply starving. 32 | I've had a most amazing time.' He reached out his hand for a cigar, and cut the end. `But come 33 | into the smoking-room. It's too long a story to tell over greasy plates.' And ringing the bell in 34 | passing, he led the way into the adjoining room. 35 | `You have told Blank, and Dash, and Chose about the machine?' he said to me, leaning back in 36 | his easy-chair and naming the three new guests. 37 | `But the thing's a mere paradox,' said the Editor. 38 | `I can't argue to-night. I don't mind telling you the story, but I can't argue. I will,' he went on, `tell 39 | you the story of what has happened to me, if you like, but you must refrain from interruptions. I 40 | want to tell it. Badly. Most of it will sound like lying. So be it! It's true--every word of it, all the 41 | same. I was in my laboratory at four o'clock, and since then . . . I've lived eight days . . . such 42 | days as no human being ever lived before! I'm nearly worn out, but I shan't sleep till I've told this 43 | thing over to you. Then I shall go to bed. But no interruptions! Is it agreed?' 44 | 9 / 49 45 | Full Text Archive 46 | https://www.fulltextarchive.com`Agreed,' said the Editor, and the rest of us echoed `Agreed.' And with that the Time Traveller 47 | began his story as I have set it forth. He sat back in his chair at first, and spoke like a weary 48 | man. Afterwards he got more animated. In writing it down I feel with only too much keenness the 49 | inadequacy of pen and ink --and, above all, my own inadequacy--to express its quality. You 50 | read, I will suppose, attentively enough; but you cannot see the speaker's white, sincere face in 51 | the bright circle of the little lamp, nor hear the intonation of his voice. You cannot know how his 52 | expression followed the turns of his story! Most of us hearers were in shadow, for the candles in 53 | the smoking-room had not been lighted, and only the face of the Journalist and the legs of the 54 | Silent Man from the knees downward were illuminated. At first we glanced now and again at 55 | each other. After a time we ceased to do that, and looked only at the Time Traveller's face. 56 | III 57 | `I told some of you last Thursday of the principles of the Time Machine, and showed you the 58 | actual thing itself, incomplete in the workshop. There it is now, a little travel-worn, truly; and one 59 | of the ivory bars is cracked, and a brass rail bent; but the rest of it's sound enough. I expected 60 | to finish it on Friday, but on Friday, when the putting together was nearly done, I found that one 61 | of the nickel bars was exactly one inch too short, and this I had to get remade; so that the thing 62 | was not complete until this morning. It was at ten o'clock to-day that the first of all Time 63 | Machines began its career. I gave it a last tap, tried all the screws again, put one more drop of 64 | oil on the quartz rod, and sat myself in the saddle. I suppose a suicide who holds a pistol to his 65 | skull feels much the same wonder at what will come next as I felt then. I took the starting lever 66 | in one hand and the stopping one in the other, pressed the first, and almost immediately the 67 | second. I seemed to reel; I felt a nightmare sensation of falling; and, looking round, I saw the 68 | laboratory exactly as before. Had anything happened? For a moment I suspected that my 69 | intellect had tricked me. Then I noted the clock. A moment before, as it seemed, it had stood at 70 | a minute or so past ten; now it was nearly half-past three! 71 | `I drew a breath, set my teeth, gripped the starting lever with both hands, and went off with a 72 | thud. The laboratory got hazy and went dark. Mrs. Watchett came in and walked, apparently 73 | without seeing me, towards the garden door. I suppose it took her a minute or so to traverse the 74 | place, but to me she seemed to shoot across the room like a rocket. I pressed the lever over to 75 | its extreme position. The night came like the turning out of a lamp, and in another moment came 76 | to-morrow. The laboratory grew faint and hazy, then fainter and ever fainter. To-morrow night 77 | came black, then day again, night again, day again, faster and faster still. An eddying murmur 78 | filled my ears, and a strange, dumb confusedness descended on my mind. 79 | `I am afraid I cannot convey the peculiar sensations of time travelling. They are excessively 80 | unpleasant. There is a feeling exactly like that one has upon a switchback--of a helpless 81 | headlong motion! I felt the same horrible anticipation, too, of an imminent smash. As I put on 82 | pace, night followed day like the flapping of a black wing. The dim suggestion of the laboratory 83 | seemed presently to fall away from me, and I saw the sun hopping swiftly across the sky, 84 | leaping it every minute, and every minute marking a day. I supposed the laboratory had been 85 | destroyed and I had come into the open air. I had a dim impression of scaffolding, but I was 86 | already going too fast to be conscious of any moving things. The slowest snail that ever crawled 87 | dashed by too fast for me. The twinkling succession of darkness and light was excessively 88 | painful to the eye. Then, in the intermittent darknesses, I saw the moon spinning swiftly through 89 | her quarters from new to full, and had a faint glimpse of the circling stars. Presently, as I went 90 | on, still gaining velocity, the palpitation of night and day merged into one continuous greyness; 91 | 10 / 49 92 | Full Text Archive 93 | https://www.fulltextarchive.comthe sky took on a wonderful deepness of blue, a splendid luminous color like that of early 94 | twilight; the jerking sun became a streak of fire, a brilliant arch, in space; the moon a fainter 95 | fluctuating band; and I could see nothing of the stars, save now and then a brighter circle 96 | flickering in the blue. 97 | `The landscape was misty and vague. I was still on the hill-side upon which this house now 98 | stands, and the shoulder rose above me grey and dim. I saw trees growing and changing like 99 | puffs of vapour, now brown, now green; they grew, spread, shivered, and passed away. I saw 100 | huge buildings rise up faint and fair, and pass like dreams. The whole surface of the earth 101 | seemed changed--melting and flowing under my eyes. The little hands upon the dials that 102 | registered my speed raced round faster and faster. Presently I noted that the sun belt swayed 103 | up and down, from solstice to solstice, in a minute or less, and that consequently my pace was 104 | over a year a minute; and minute by minute the white snow flashed across the world, and 105 | vanished, and was followed by the bright, brief green of spring. 106 | `The unpleasant sensations of the start were less poignant now. They merged at last into a kind 107 | of hysterical exhilaration. I remarked indeed a clumsy swaying of the machine, for which I was 108 | unable to account. But my mind was too confused to attend to it, so with a kind of madness 109 | growing upon me, I flung myself into futurity. At first I scarce thought of stopping, scarce thought 110 | of anything but these new sensations. But presently a fresh series of impressions grew up in my 111 | mind--a certain curiosity and therewith a certain dread--until at last they took complete 112 | possession of me. What strange developments of humanity, what wonderful advances upon our 113 | rudimentary civilization, I thought, might not appear when I came to look nearly into the dim 114 | elusive world that raced and fluctuated before my eyes! I saw great and splendid architecture 115 | rising about me, more massive than any buildings of our own time, and yet, as it seemed, built 116 | of glimmer and mist. I saw a richer green flow up the hill-side, and remain there, without any 117 | wintry intermission. Even through the veil of my confusion the earth seemed very fair. And so 118 | my mind came round to the business of stopping, 119 | `The peculiar risk lay in the possibility of my finding some substance in the space which I, or the 120 | machine, occupied. So long as I travelled at a high velocity through time, this scarcely mattered; 121 | I was, so to speak, attenuated--was slipping like a vapour through the interstices of intervening 122 | substances! But to come to a stop involved the jamming of myself, molecule by molecule, into 123 | whatever lay in my way; meant bringing my atoms into such intimate contact with those of the 124 | obstacle that a profound chemical reaction--possibly a far-reaching explosion --would result, and 125 | blow myself and my apparatus out of all possible dimensions--into the Unknown. This possibility 126 | had occurred to me again and again while I was making the machine; but then I had cheerfully 127 | accepted it as an unavoidable risk-- one of the risks a man has got to take! Now the risk was 128 | inevitable, I no longer saw it in the same cheerful light. The fact is that insensibly, the absolute 129 | strangeness of everything, the sickly jarring and swaying of the machine, above all, the feeling 130 | of prolonged falling, had absolutely upset my nerve. I told myself that I could never stop, and 131 | with a gust of petulance I resolved to stop forthwith. Like an impatient fool, I lugged over the 132 | lever, and incontinently the thing went reeling over, and I was flung headlong through the air. 133 | `There was the sound of a clap of thunder in my ears. I may have been stunned for a moment. 134 | A pitiless hail was hissing round me, and I was sitting on soft turf in front of the overset machine. 135 | Everything still seemed grey, but presently I remarked that the confusion in my ears was gone. I 136 | looked round me. I was on what seemed to be a little lawn in a garden, surrounded by 137 | rhododendron bushes, and I noticed that their mauve and purple blossoms were dropping in a 138 | 11 / 49 139 | Full Text Archive 140 | https://www.fulltextarchive.comshower under the beating of the hail-stones. The rebounding, dancing hail hung in a cloud over 141 | the machine, and drove along the ground like smoke. In a moment I was wet to the skin. "Fine 142 | hospitality," said I, "to a man who has travelled innumerable years to see you." 143 | `Presently I thought what a fool I was to get wet. I stood up and looked round me. A colossal 144 | figure, carved apparently in some white stone, loomed indistinctly beyond the rhododendrons 145 | through the hazy downpour. But all else of the world was invisible. 146 | `My sensations would be hard to describe. As the columns of hail grew thinner, I saw the white 147 | figure more distinctly. It was very large, for a silver birch-tree touched its shoulder. It was of 148 | white marble, in shape something like a winged sphinx, but the wings, instead of being carried 149 | vertically at the sides, were spread so that it seemed to hover. The pedestal, it appeared to me, 150 | was of bronze, and was thick with verdigris. It chanced that the face was towards me; the 151 | sightless eyes seemed to watch me; there was the faint shadow of a smile on the lips. It was 152 | greatly weather-worn, and that imparted an unpleasant suggestion of disease. I stood looking at 153 | it for a little space--half a minute, perhaps, or half an hour. It seemed to advance and to recede 154 | as the hail drove before it denser or thinner. At last I tore my eyes from it for a moment and saw 155 | that the hail curtain had worn threadbare, and that the sky was lightening with the promise of 156 | the Sun. 157 | `I looked up again at the crouching white shape, and the full temerity of my voyage came 158 | suddenly upon me. What might appear when that hazy curtain was altogether withdrawn? What 159 | might not have happened to men? What if cruelty had grown into a common passion? What if in 160 | this interval the race had lost its manliness and had developed into something inhuman, 161 | unsympathetic, and overwhelmingly powerful? I might seem some old-world savage animal, 162 | only the more dreadful and disgusting for our common likeness--a foul creature to be 163 | incontinently slain. 164 | `Already I saw other vast shapes--huge buildings with intricate parapets and tall columns, with a 165 | wooded hill-side dimly creeping in upon me through the lessening storm. I was seized with a 166 | panic fear. I turned frantically to the Time Machine, and strove hard to readjust it. As I did so the 167 | shafts of the sun smote through the thunderstorm. The grey downpour was swept aside and 168 | vanished like the trailing garments of a ghost. Above me, in the intense blue of the summer sky, 169 | some faint brown shreds of cloud whirled into nothingness. The great buildings about me stood 170 | out clear and distinct, shining with the wet of the thunderstorm, and picked out in white by the 171 | unmelted hailstones piled along their courses. I felt naked in a strange world. I felt as perhaps a 172 | bird may feel in the clear air, knowing the hawk wings above and will swoop. My fear grew to 173 | frenzy. I took a breathing space, set my teeth, and again grappled fiercely, wrist and knee, with 174 | the machine. It gave under my desperate onset and turned over. It struck my chin violently. One 175 | hand on the saddle, the other on the lever, I stood panting heavily in attitude to mount again. 176 | `But with this recovery of a prompt retreat my courage recovered. I looked more curiously and 177 | less fearfully at this world of the remote future. In a circular opening, high up in the wall of the 178 | nearer house, I saw a group of figures clad in rich soft robes. They had seen me, and their 179 | faces were directed towards me. 180 | `Then I heard voices approaching me. Coming through the bushes by the White Sphinx were 181 | the heads and shoulders of men running. One of these emerged in a pathway leading straight to 182 | the little lawn upon which I stood with my machine. He was a slight creature--perhaps four feet 183 | 12 / 49 184 | Full Text Archive 185 | https://www.fulltextarchive.comhigh--clad in a purple tunic, girdled at the waist with a leather belt. Sandals or buskins--I could 186 | not clearly distinguish which--were on his feet; his legs were bare to the knees, and his head 187 | was bare. Noticing that, I noticed for the first time how warm the air was. 188 | `He struck me as being a very beautiful and graceful creature, but indescribably frail. His flushed 189 | face reminded me of the more beautiful kind of consumptive--that hectic beauty of which we 190 | used to hear so much. At the sight of him I suddenly regained confidence. I took my hands from 191 | the machine. 192 | IV 193 | `In another moment we were standing face to face, I and this fragile thing out of futurity. He 194 | came straight up to me and laughed into my eyes. The absence from his bearing of any sign of 195 | fear struck me at once. Then he turned to the two others who were following him and spoke to 196 | them in a strange and very sweet and liquid tongue. 197 | `There were others coming, and presently a little group of perhaps eight or ten of these 198 | exquisite creatures were about me. One of them addressed me. It came into my head, oddly 199 | enough, that my voice was too harsh and deep for them. So I shook my head, and, pointing to 200 | my ears, shook it again. He came a step forward, hesitated, and then touched my hand. Then I 201 | felt other soft little tentacles upon my back and shoulders. They wanted to make sure I was real. 202 | There was nothing in this at all alarming. Indeed, there was something in these pretty little 203 | people that inspired confidence--a graceful gentleness, a certain childlike ease. And besides, 204 | they looked so frail that I could fancy myself flinging the whole dozen of them about like nine- 205 | pins. But I made a sudden motion to warn them when I saw their little pink hands feeling at the 206 | Time Machine. Happily then, when it was not too late, I thought of a danger I had hitherto 207 | forgotten, and reaching over the bars of the machine I unscrewed the little levers that would set 208 | it in motion, and put these in my pocket. Then I turned again to see what I could do in the way of 209 | communication. 210 | `And then, looking more nearly into their features, I saw some further peculiarities in their 211 | Dresden-china type of prettiness. Their hair, which was uniformly curly, came to a sharp end at 212 | the neck and cheek; there was not the faintest suggestion of it on the face, and their ears were 213 | singularly minute. The mouths were small, with bright red, rather thin lips, and the little chins ran 214 | to a point. The eyes were large and mild; and--this may seem egotism on my part--I fancied 215 | even that there was a certain lack of the interest I might have expected in them. 216 | `As they made no effort to communicate with me, but simply stood round me smiling and 217 | speaking in soft cooing notes to each other, I began the conversation. I pointed to the Time 218 | Machine and to myself. Then hesitating for a moment how to express time, I pointed to the sun. 219 | At once a quaintly pretty little figure in chequered purple and white followed my gesture, and 220 | then astonished me by imitating the sound of thunder. 221 | `For a moment I was staggered, though the import of his gesture was plain enough. The 222 | question had come into my mind abruptly: were these creatures fools? You may hardly 223 | understand how it took me. You see I had always anticipated that the people of the year Eight 224 | Hundred and Two Thousand odd would be incredibly in front of us in knowledge, art, everything. 225 | Then one of them suddenly asked me a question that showed him to be on the intellectual level 226 | of one of our five-year-old children-- asked me, in fact, if I had come from the sun in a 227 | 13 / 49 228 | Full Text Archive 229 | https://www.fulltextarchive.comthunderstorm! It let loose the judgment I had suspended upon their clothes, their frail light limbs, 230 | and fragile features. A flow of disappointment rushed across my mind. For a moment I felt that I 231 | had built the Time Machine in vain. 232 | `I nodded, pointed to the sun, and gave them such a vivid rendering of a thunderclap as startled 233 | them. They all withdrew a pace or so and bowed. Then came one laughing towards me, 234 | carrying a chain of beautiful flowers altogether new to me, and put it about my neck. The idea 235 | was received with melodious applause; and presently they were all running to and fro for 236 | flowers, and laughingly flinging them upon me until I was almost smothered with blossom. You 237 | who have never seen the like can scarcely imagine what delicate and wonderful flowers 238 | countless years of culture had created. Then someone suggested that their plaything should be 239 | exhibited in the nearest building, and so I was led past the sphinx of white marble, which had 240 | seemed to watch me all the while with a smile at my astonishment, towards a vast grey edifice 241 | of fretted stone. As I went with them the memory of my confident anticipations of a profoundly 242 | grave and intellectual posterity came, with irresistible merriment, to my mind. 243 | `The building had a huge entry, and was altogether of colossal dimensions. I was naturally most 244 | occupied with the growing crowd of little people, and with the big open portals that yawned 245 | before me shadowy and mysterious. My general impression of the world I saw over their heads 246 | was a tangled waste of beautiful bushes and flowers, a long neglected and yet weedless 247 | garden. I saw a number of tall spikes of strange white flowers, measuring a foot perhaps across 248 | the spread of the waxen petals. They grew scattered, as if wild, among the variegated shrubs, 249 | but, as I say, I did not examine them closely at this time. The Time Machine was left deserted 250 | on the turf among the rhododendrons. 251 | `The arch of the doorway was richly carved, but naturally I did not observe the carving very 252 | narrowly, though I fancied I saw suggestions of old Phoenician decorations as I passed through, 253 | and it struck me that they were very badly broken and weather- worn. Several more brightly clad 254 | people met me in the doorway, and so we entered, I, dressed in dingy nineteenth-century 255 | garments, looking grotesque enough, garlanded with flowers, and surrounded by an eddying 256 | mass of bright, soft-colored robes and shining white limbs, in a melodious whirl of laughter and 257 | laughing speech. 258 | `The big doorway opened into a proportionately great hall hung with brown. The roof was in 259 | shadow, and the windows, partially glazed with coloured glass and partially unglazed, admitted 260 | a tempered light. The floor was made up of huge blocks of some very hard white metal, not 261 | plates nor slabs--blocks, and it was so much worn, as I judged by the going to and fro of past 262 | generations, as to be deeply channelled along the more frequented ways. Transverse to the 263 | length were innumerable tables made of slabs of polished stone, raised perhaps a foot from the 264 | floor, and upon these were heaps of fruits. Some I recognized as a kind of hypertrophied 265 | raspberry and orange, but for the most part they were strange. 266 | `Between the tables was scattered a great number of cushions. Upon these my conductors 267 | seated themselves, signing for me to do likewise. With a pretty absence of ceremony they 268 | began to eat the fruit with their hands, flinging peel and stalks, and so forth, into the round 269 | openings in the sides of the tables. I was not loath to follow their example, for I felt thirsty and 270 | hungry. As I did so I surveyed the hall at my leisure. 271 | `And perhaps the thing that struck me most was its dilapidated look. The stained-glass windows, 272 | 14 / 49 273 | Full Text Archive 274 | https://www.fulltextarchive.comwhich displayed only a geometrical pattern, were broken in many places, and the curtains that 275 | hung across the lower end were thick with dust. And it caught my eye that the corner of the 276 | marble table near me was fractured. Nevertheless, the general effect was extremely rich and 277 | picturesque. There were, perhaps, a couple of hundred people dining in the hall, and most of 278 | them, seated as near to me as they could come, were watching me with interest, their little eyes 279 | shining over the fruit they were eating. All were clad in the same soft and yet strong, silky 280 | material. 281 | `Fruit, by the by, was all their diet. These people of the remote future were strict vegetarians, 282 | and while I was with them, in spite of some carnal cravings, I had to be frugivorous also. Indeed, 283 | I found afterwards that horses, cattle, sheep, dogs, had followed the Ichthyosaurus into 284 | extinction. But the fruits were very delightful; one, in particular, that seemed to be in season all 285 | the time I was there--a floury thing in a three-sided husk --was especially good, and I made it my 286 | staple. At first I was puzzled by all these strange fruits, and by the strange flowers I saw, but 287 | later I began to perceive their import. 288 | `However, I am telling you of my fruit dinner in the distant future now. So soon as my appetite 289 | was a little checked, I determined to make a resolute attempt to learn the speech of these new 290 | men of mine. Clearly that was the next thing to do. The fruits seemed a convenient thing to 291 | begin upon, and holding one of these up I began a series of interrogative sounds and gestures. 292 | I had some considerable difficulty in conveying my meaning. At first my efforts met with a stare 293 | of surprise or inextinguishable laughter, but presently a fair-haired little creature seemed to 294 | grasp my intention and repeated a name. They had to chatter and explain the business at great 295 | length to each other, and my first attempts to make the exquisite little sounds of their language 296 | caused an immense amount of amusement. However, I felt like a schoolmaster amidst children, 297 | and persisted, and presently I had a score of noun substantives at least at my command; and 298 | then I got to demonstrative pronouns, and even the verb "to eat." But it was slow work, and the 299 | little people soon tired and wanted to get away from my interrogations, so I determined, rather 300 | of necessity, to let them give their lessons in little doses when they felt inclined. And very little 301 | doses I found they were before long, for I never met people more indolent or more easily 302 | fatigued. 303 | `A queer thing I soon discovered about my little hosts, and that was their lack of interest. They 304 | would come to me with eager cries of astonishment, like children, but like children they would 305 | soon stop examining me and wander away after some other toy. The dinner and my 306 | conversational beginnings ended, I noted for the first time that almost all those who had 307 | surrounded me at first were gone. It is odd, too, how speedily I came to disregard these little 308 | people. I went out through the portal into the sunlit world again as soon as my hunger was 309 | satisfied. I was continually meeting more of these men of the future, who would follow me a little 310 | distance, chatter and laugh about me, and, having smiled and gesticulated in a friendly way, 311 | leave me again to my own devices. 312 | `The calm of evening was upon the world as I emerged from the great hall, and the scene was 313 | lit by the warm glow of the setting sun. At first things were very confusing. Everything was so 314 | entirely different from the world I had known--even the flowers. The big building I had left was 315 | situated on the slope of a broad river valley, but the Thames had shifted perhaps a mile from its 316 | present position. I resolved to mount to the summit of a crest perhaps a mile and a half away, 317 | from which I could get a wider view of this our planet in the year Eight Hundred and Two 318 | Thousand Seven Hundred and One A.D. For that, I should explain, was the date the little dials 319 | 15 / 49 320 | Full Text Archive 321 | https://www.fulltextarchive.comof my machine recorded. 322 | `As I walked I was watching for every impression that could possibly help to explain the 323 | condition of ruinous splendour in which I found the world--for ruinous it was. A little way up the 324 | hill, for instance, was a great heap of granite, bound together by masses of aluminium, a vast 325 | labyrinth of precipitous walls and crumpled heaps, amidst which were thick heaps of very 326 | beautiful pagoda-like plants--nettles possibly--but wonderfully tinted with brown about the 327 | leaves, and incapable of stinging. It was evidently the derelict remains of some vast structure, to 328 | what end built I could not determine. It was here that I was destined, at a later date, to have a 329 | very strange experience--the first intimation of a still stranger discovery--but of that I will speak 330 | in its proper place. 331 | `Looking round with a sudden thought, from a terrace on which I rested for a while, I realized 332 | that there were no small houses to be seen. Apparently the single house, and possibly even the 333 | household, had vanished. Here and there among the greenery were palace-like buildings, but 334 | the house and the cottage, which form such characteristic features of our own English 335 | landscape, had disappeared. 336 | `"Communism," said I to myself. 337 | `And on the heels of that came another thought. I looked at the half-dozen little figures that were 338 | following me. Then, in a flash, I perceived that all had the same form of costume, the same soft 339 | hairless visage, and the same girlish rotundity of limb. It may seem strange, perhaps, that I had 340 | not noticed this before. But everything was so strange. Now, I saw the fact plainly enough. In 341 | costume, and in all the differences of texture and bearing that now mark off the sexes from each 342 | other, these people of the future were alike. And the children seemed to my eyes to be but the 343 | miniatures of their parents. I judged, then, that the children of that time were extremely 344 | precocious, physically at least, and I found afterwards abundant verification of my opinion. 345 | `Seeing the ease and security in which these people were living, I felt that this close 346 | resemblance of the sexes was after all what one would expect; for the strength of a man and 347 | the softness of a woman, the institution of the family, and the differentiation of occupations are 348 | mere militant necessities of an age of physical force; where population is balanced and 349 | abundant, much childbearing becomes an evil rather than a blessing to the State; where 350 | violence comes but rarely and off-spring are secure, there is less necessity--indeed there is no 351 | necessity--for an efficient family, and the specialization of the sexes with reference to their 352 | children's needs disappears. We see some beginnings of this even in our own time, and in this 353 | future age it was complete. This, I must remind you, was my speculation at the time. Later, I 354 | was to appreciate how far it fell short of the reality. 355 | `While I was musing upon these things, my attention was attracted by a pretty little structure, 356 | like a well under a cupola. I thought in a transitory way of the oddness of wells still existing, and 357 | then resumed the thread of my speculations. There were no large buildings towards the top of 358 | the hill, and as my walking powers were evidently miraculous, I was presently left alone for the 359 | first time. With a strange sense of freedom and adventure I pushed on up to the crest. 360 | `There I found a seat of some yellow metal that I did not recognize, corroded in places with a 361 | kind of pinkish rust and half smothered in soft moss, the arm-rests cast and filed into the 362 | resemblance of griffins' heads. I sat down on it, and I surveyed the broad view of our old world 363 | Full Text Archive 364 | https://www.fulltextarchive.comindifferent and unseeing. The black tunnel yawned indefinitely off at a steep descending grade, 365 | its aperture adorned with grotesquely chiselled jambs and lintel. From that cryptical mouth we 366 | fancied a current of slightly warmer air and perhaps even a suspicion of vapour proceeded; and 367 | we wondered what living entities other than penguins the limitless void below, and the 368 | contiguous honeycombings of the land and the titan mountains, might conceal. We wondered, 369 | too, whether the trace of mountain-top smoke at first suspected by poor Lake, as well as the 370 | odd haze we had ourselves perceived around the rampart-crowned peak, might not be caused 371 | by the tortuous-channelled rising of some such vapour from the unfathomed regions of earth’s 372 | core. 373 | Entering the tunnel, we saw that its outline was—at least at the start—about fifteen feet each way; 374 | sides, floor, and arched roof composed of the usual megalithic masonry. The sides were 375 | sparsely decorated with cartouches of conventional designs in a late, decadent style; and all the 376 | construction and carving were marvellously well preserved. The floor was quite clear, except for 377 | a slight detritus bearing outgoing penguin tracks and the inward tracks of those others. The 378 | farther one advanced, the warmer it became; so that we were soon unbuttoning our heavy 379 | garments. We wondered whether there were any actually igneous manifestations below, and 380 | whether the waters of that sunless sea were hot. After a short distance the masonry gave place 381 | to solid rock, though the tunnel kept the same proportions and presented the same aspect of 382 | carved regularity. Occasionally its varying grade became so steep that grooves were cut in the 383 | floor. Several times we noted the mouths of small lateral galleries not recorded in our diagrams; 384 | none of them such as to complicate the problem of our return, and all of them welcome as 385 | possible refuges in case we met unwelcome entities on their way back from the abyss. The 386 | nameless scent of such things was very distinct. Doubtless it was suicidally foolish to venture 387 | into that tunnel under the known conditions, but the lure of the unplumbed is stronger in certain 388 | persons than most suspect—indeed, it was just such a lure which had brought us to this 389 | unearthly polar waste in the first place. We saw several penguins as we passed along, and 390 | speculated on the distance we would have to traverse. The carvings had led us to expect a 391 | steep downhill walk of about a mile to the abyss, but our previous wanderings had shewn us 392 | that matters of scale were not wholly to be depended on. 393 | After about a quarter of a mile that nameless scent became greatly accentuated, and we kept 394 | very careful track of the various lateral openings we passed. There was no visible vapour as at 395 | the mouth, but this was doubtless due to the lack of contrasting cooler air. The temperature was 396 | rapidly ascending, and we were not surprised to come upon a careless heap of material 397 | shudderingly familiar to us. It was composed of furs and tent-cloth taken from Lake’s camp, and 398 | we did not pause to study the bizarre forms into which the fabrics had been slashed. Slightly 399 | beyond this point we noticed a decided increase in the size and number of the side-galleries, 400 | and concluded that the densely honeycombed region beneath the higher foothills must now 401 | have been reached. The nameless scent was now curiously mixed with another and scarcely 402 | less offensive odour—of what nature we could not guess, though we thought of decaying 403 | organisms and perhaps unknown subterrene fungi. Then came a startling expansion of the 404 | tunnel for which the carvings had not prepared us—a broadening and rising into a lofty, natural- 405 | looking elliptical cavern with a level floor; some 75 feet long and 50 broad, and with many 406 | immense side-passages leading away into cryptical darkness. 407 | 63 / 74 408 | Full Text Archive 409 | https://www.fulltextarchive.comThough this cavern was natural in appearance, an inspection with both torches suggested that it 410 | had been formed by the artificial destruction of several walls between adjacent honeycombings. 411 | The walls were rough, and the high vaulted roof was thick with stalactites; but the solid rock 412 | floor had been smoothed off, and was free from all debris, detritus, or even dust to a positively 413 | abnormal extent. Except for the avenue through which we had come, this was true of the floors 414 | of all the great galleries opening off from it; and the singularity of the condition was such as to 415 | set us vainly puzzling. The curious new foetor which had supplemented the nameless scent was 416 | excessively pungent here; so much so that it destroyed all trace of the other. Something about 417 | this whole place, with its polished and almost glistening floor, struck us as more vaguely baffling 418 | and horrible than any of the monstrous things we had previously encountered. 419 | The regularity of the passage immediately ahead, as well as the larger proportion of penguin- 420 | droppings there, prevented all confusion as to the right course amidst this plethora of equally 421 | great cave-mouths. Nevertheless we resolved to resume our paper trail-blazing if any further 422 | complexity should develop; for dust tracks, of course, could no longer be expected. Upon 423 | resuming our direct progress we cast a beam of torchlight over the tunnel walls—and stopped 424 | short in amazement at the supremely radical change which had come over the carvings in this 425 | part of the passage. We realised, of course, the great decadence of the Old Ones’ sculpture at 426 | the time of the tunnelling; and had indeed noticed the inferior workmanship of the arabesques in 427 | the stretches behind us. But now, in this deeper section beyond the cavern, there was a sudden 428 | difference wholly transcending explanation—a difference in basic nature as well as in mere 429 | quality, and involving so profound and calamitous a degradation of skill that nothing in the 430 | hitherto observed rate of decline could have led one to expect it. 431 | This new and degenerate work was coarse, bold, and wholly lacking in delicacy of detail. It was 432 | counter-sunk with exaggerated depth in bands following the same general line as the sparse 433 | cartouches of the earlier sections, but the height of the reliefs did not reach the level of the 434 | general surface. Danforth had the idea that it was a second carving—a sort of palimpsest formed 435 | after the obliteration of a previous design. In nature it was wholly decorative and conventional; 436 | and consisted of crude spirals and angles roughly following the quintile mathematical tradition of 437 | the Old Ones, yet seeming more like a parody than a perpetuation of that tradition. We could 438 | not get it out of our minds that some subtly but profoundly alien element had been added to the 439 | aesthetic feeling behind the technique—an alien element, Danforth guessed, that was 440 | responsible for the manifestly laborious substitution. It was like, yet disturbingly unlike, what we 441 | had come to recognise as the Old Ones’ art; and I was persistently reminded of such hybrid 442 | things as the ungainly Palmyrene sculptures fashioned in the Roman manner. That others had 443 | recently noticed this belt of carving was hinted by the presence of a used torch battery on the 444 | floor in front of one of the most characteristic designs. 445 | Since we could not afford to spend any considerable time in study, we resumed our advance 446 | after a cursory look; though frequently casting beams over the walls to see if any further 447 | 64 / 74 448 | Full Text Archive 449 | https://www.fulltextarchive.comso, where were they? Were they close at hand and likely to form an immediate menace to us? 450 | We glanced anxiously at some of the smooth-floored lateral passages as we continued our slow 451 | and frankly reluctant approach. Whatever the conflict was, it had clearly been that which had 452 | frightened the penguins into their unaccustomed wandering. It must, then, have arisen near that 453 | faintly heard rookery in the incalculable gulf beyond, since there were no signs that any birds 454 | had normally dwelt here. Perhaps, we reflected, there had been a hideous running fight, with 455 | the weaker party seeking to get back to the cached sledges when their pursuers finished them. 456 | One could picture the daemoniac fray between namelessly monstrous entities as it surged out 457 | of the black abyss with great clouds of frantic penguins squawking and scurrying ahead. 458 | I say that we approached those sprawling and incomplete obstructions slowly and reluctantly. 459 | Would to heaven we had never approached them at all, but had run back at top speed out of 460 | that blasphemous tunnel with the greasily smooth floors and the degenerate murals aping and 461 | mocking the things they had superseded—run back, before we had seen what we did see, and 462 | before our minds were burned with something which will never let us breathe easily again! 463 | Both of our torches were turned on the prostrate objects, so that we soon realised the dominant 464 | factor in their incompleteness. Mauled, compressed, twisted, and ruptured as they were, their 465 | chief common injury was total decapitation. From each one the tentacled starfish-head had 466 | been removed; and as we drew near we saw that the manner of removal looked more like some 467 | hellish tearing or suction than like any ordinary form of cleavage. Their noisome dark-green 468 | ichor formed a large, spreading pool; but its stench was half overshadowed by that newer and 469 | stranger stench, here more pungent than at any other point along our route. Only when we had 470 | come very close to the sprawling obstructions could we trace that second, unexplainable foetor 471 | to any immediate source—and the instant we did so Danforth, remembering certain very vivid 472 | sculptures of the Old Ones’ history in the Permian age 150 million years ago, gave vent to a 473 | nerve-tortured cry which echoed hysterically through that vaulted and archaic passage with the 474 | evil palimpsest carvings. 475 | I came only just short of echoing his cry myself; for I had seen those primal sculptures, too, and 476 | had shudderingly admired the way the nameless artist had suggested that hideous slime- 477 | coating found on certain incomplete and prostrate Old Ones—those whom the frightful shoggoths 478 | had characteristically slain and sucked to a ghastly headlessness in the great war of re- 479 | subjugation. They were infamous, nightmare sculptures even when telling of age-old, bygone 480 | things; for shoggoths and their work ought not to be seen by human beings or portrayed by any 481 | beings. The mad author of the Necronomicon had nervously tried to swear that none had been 482 | bred on this planet, and that only drugged dreamers had ever conceived them. Formless 483 | protoplasm able to mock and reflect all forms and organs and processes—viscous agglutinations 484 | of bubbling cells—rubbery fifteen-foot spheroids infinitely plastic and ductile—slaves of 485 | suggestion, builders of cities—more and more sullen, more and more intelligent, more and more 486 | amphibious, more and more imitative—Great God! What madness made even those 487 | blasphemous Old Ones willing to use and to carve such things? 488 | 66 / 74 489 | Full Text Archive 490 | https://www.fulltextarchive.comThe new sound, as I have intimated, upset much that we had decided; because it was what 491 | poor Lake’s dissection had led us to attribute to those we had just judged dead. It was, 492 | Danforth later told me, precisely what he had caught in infinitely muffled form when at that spot 493 | beyond the alley-corner above the glacial level; and it certainly had a shocking resemblance to 494 | the wind-pipings we had both heard around the lofty mountain caves. At the risk of seeming 495 | puerile I will add another thing, too; if only because of the surprising way Danforth’s impression 496 | chimed with mine. Of course common reading is what prepared us both to make the 497 | interpretation, though Danforth has hinted at queer notions about unsuspected and forbidden 498 | sources to which Poe may have had access when writing his Arthur Gordon Pym a century ago. 499 | It will be remembered that in that fantastic tale there is a word of unknown but terrible and 500 | prodigious significance connected with the antarctic and screamed eternally by the gigantic, 501 | spectrally snowy birds of that malign region’s core. “Tekeli-li! Tekeli-li!” That, I may admit, is 502 | exactly what we thought we heard conveyed by that sudden sound behind the advancing white 503 | mist—that insidious musical piping over a singularly wide range. 504 | We were in full flight before three notes or syllables had been uttered, though we knew that the 505 | swiftness of the Old Ones would enable any scream-roused and pursuing survivor of the 506 | slaughter to overtake us in a moment if it really wished to do so. We had a vague hope, 507 | however, that non-aggressive conduct and a display of kindred reason might cause such a 508 | being to spare us in case of capture; if only from scientific curiosity. After all, if such an one had 509 | nothing to fear for itself it would have no motive in harming us. Concealment being futile at this 510 | juncture, we used our torch for a running glance behind, and perceived that the mist was 511 | thinning. Would we see, at last, a complete and living specimen of those others? Again came 512 | that insidious musical piping—“Tekeli-li! Tekeli-li!” 513 | Then, noting that we were actually gaining on our pursuer, it occurred to us that the entity might 514 | be wounded. We could take no chances, however, since it was very obviously approaching in 515 | answer to Danforth’s scream rather than in flight from any other entity. The timing was too 516 | close to admit of doubt. Of the whereabouts of that less conceivable and less mentionable 517 | nightmare—that foetid, unglimpsed mountain of slime-spewing protoplasm whose race had 518 | conquered the abyss and sent land pioneers to re-carve and squirm through the burrows of the 519 | hills—we could form no guess; and it cost us a genuine pang to leave this probably crippled Old 520 | One—perhaps a lone survivor—to the peril of recapture and a nameless fate. 521 | Thank heaven we did not slacken our run. The curling mist had thickened again, and was 522 | driving ahead with increased speed; whilst the straying penguins in our rear were squawking 523 | and screaming and displaying signs of a panic really surprising in view of their relatively minor 524 | confusion when we had passed them. Once more came that sinister, wide-ranged 525 | piping—“Tekeli-li! Tekeli-li!” We had been wrong. The thing was not wounded, but had merely 526 | paused on encountering the bodies of its fallen kindred and the hellish slime inscription above 527 | them. We could never know what that daemon message was—but those burials at Lake’s camp 528 | had shewn how much importance the beings attached to their dead. Our recklessly used torch 529 | now revealed ahead of us the large open cavern where various ways converged, and we were 530 | 68 / 74 531 | Full Text Archive 532 | https://www.fulltextarchive.comswept so evilly free of all litter. Still came that eldritch, mocking cry—“Tekeli-li! Tekeli-li!” And at 533 | last we remembered that the daemoniac shoggoths—given life, thought, and plastic organ 534 | patterns solely by the Old Ones, and having no language save that which the dot-groups 535 | expressed—had likewise no voice save the imitated accents of their bygone masters. 536 | XII. 537 | Danforth and I have recollections of emerging into the great sculptured hemisphere and of 538 | threading our back trail through the Cyclopean rooms and corridors of the dead city; yet these 539 | are purely dream-fragments involving no memory of volition, details, or physical exertion. It was 540 | as if we floated in a nebulous world or dimension without time, causation, or orientation. The 541 | grey half-daylight of the vast circular space sobered us somewhat; but we did not go near those 542 | cached sledges or look again at poor Gedney and the dog. They have a strange and titanic 543 | mausoleum, and I hope the end of this planet will find them still undisturbed. 544 | It was while struggling up the colossal spiral incline that we first felt the terrible fatigue and short 545 | breath which our race through the thin plateau air had produced; but not even the fear of 546 | collapse could make us pause before reaching the normal outer realm of sun and sky. There 547 | was something vaguely appropriate about our departure from those buried epochs; for as we 548 | wound our panting way up the sixty-foot cylinder of primal masonry we glimpsed beside us a 549 | continuous procession of heroic sculptures in the dead race’s early and undecayed 550 | technique—a farewell from the Old Ones, written fifty million years ago. 551 | Finally scrambling out at the top, we found ourselves on a great mound of tumbled blocks; with 552 | the curved walls of higher stonework rising westward, and the brooding peaks of the great 553 | mountains shewing beyond the more crumbled structures toward the east. The low antarctic sun 554 | of midnight peered redly from the southern horizon through rifts in the jagged ruins, and the 555 | terrible age and deadness of the nightmare city seemed all the starker by contrast with such 556 | relatively known and accustomed things as the features of the polar landscape. The sky above 557 | was a churning and opalescent mass of tenuous ice-vapours, and the cold clutched at our 558 | vitals. Wearily resting the outfit-bags to which we had instinctively clung throughout our 559 | desperate flight, we rebuttoned our heavy garments for the stumbling climb down the mound 560 | and the walk through the aeon-old stone maze to the foothills where our aëroplane waited. Of 561 | what had set us fleeing from the darkness of earth’s secret and archaic gulfs we said nothing at 562 | all. 563 | In less than a quarter of an hour we had found the steep grade to the foothills—the probable 564 | ancient terrace—by which we had descended, and could see the dark bulk of our great plane 565 | 71 / 74 566 | Full Text Archive 567 | https://www.fulltextarchive.comamidst the sparse ruins on the rising slope ahead. Half way uphill toward our goal we paused for 568 | a momentary breathing-spell, and turned to look again at the fantastic palaeogean tangle of 569 | incredible stone shapes below us—once more outlined mystically against an unknown west. As 570 | we did so we saw that the sky beyond had lost its morning haziness; the restless ice-vapours 571 | having moved up to the zenith, where their mocking outlines seemed on the point of settling into 572 | some bizarre pattern which they feared to make quite definite or conclusive. 573 | There now lay revealed on the ultimate white horizon behind the grotesque city a dim, elfin line 574 | of pinnacled violet whose needle-pointed heights loomed dream-like against the beckoning rose- 575 | colour of the western sky. Up toward this shimmering rim sloped the ancient table-land, the 576 | depressed course of the bygone river traversing it as an irregular ribbon of shadow. For a 577 | second we gasped in admiration of the scene’s unearthly cosmic beauty, and then vague 578 | horror began to creep into our souls. For this far violet line could be nothing else than the 579 | terrible mountains of the forbidden land—highest of earth’s peaks and focus of earth’s evil; 580 | harbourers of nameless horrors and Archaean secrets; shunned and prayed to by those who 581 | feared to carve their meaning; untrodden by any living thing of earth, but visited by the sinister 582 | lightnings and sending strange beams across the plains in the polar night—beyond doubt the 583 | unknown archetype of that dreaded Kadath in the Cold Waste beyond abhorrent Leng, whereof 584 | unholy primal legends hint evasively. We were the first human beings ever to see them—and I 585 | hope to God we may be the last. 586 | If the sculptured maps and pictures in that pre-human city had told truly, these cryptic violet 587 | mountains could not be much less than 300 miles away; yet none the less sharply did their dim 588 | elfin essence jut above that remote and snowy rim, like the serrated edge of a monstrous alien 589 | planet about to rise into unaccustomed heavens. Their height, then, must have been 590 | tremendous beyond all known comparison—carrying them up into tenuous atmospheric strata 591 | peopled by such gaseous wraiths as rash flyers have barely lived to whisper of after 592 | unexplainable falls. Looking at them, I thought nervously of certain sculptured hints of what the 593 | great bygone river had washed down into the city from their accursed slopes—and wondered 594 | how much sense and how much folly had lain in the fears of those Old Ones who carved them 595 | so reticently. I recalled how their northerly end must come near the coast at Queen Mary Land, 596 | where even at that moment Sir Douglas Mawson’s expedition was doubtless working less than 597 | a thousand miles away; and hoped that no evil fate would give Sir Douglas and his men a 598 | glimpse of what might lie beyond the protecting coastal range. Such thoughts formed a measure 599 | of my overwrought condition at the time—and Danforth seemed to be even worse. 600 | Yet long before we had passed the great star-shaped ruin and reached our plane our fears had 601 | become transferred to the lesser but vast enough range whose re-crossing lay ahead of us. 602 | From these foothills the black, ruin-crusted slopes reared up starkly and hideously against the 603 | east, again reminding us of those strange Asian paintings of Nicholas Roerich; and when we 604 | thought of the damnable honeycombs inside them, and of the frightful amorphous entities that 605 | might have pushed their foetidly squirming way even to the topmost hollow pinnacles, we could 606 | not face without panic the prospect of again sailing by those suggestive skyward cave-mouths 607 | 72 / 74 608 | Full Text Archive 609 | https://www.fulltextarchive.comwhere the wind made sounds like an evil musical piping over a wide range. To make matters 610 | worse, we saw distinct traces of local mist around several of the summits—as poor Lake must 611 | have done when he made that early mistake about volcanism—and thought shiveringly of that 612 | kindred mist from which we had just escaped; of that, and of the blasphemous, horror-fostering 613 | abyss whence all such vapours came. 614 | All was well with the plane, and we clumsily hauled on our heavy flying furs. Danforth got the 615 | engine started without trouble, and we made a very smooth takeoff over the nightmare city. 616 | Below us the primal Cyclopean masonry spread out as it had done when first we saw it—so 617 | short, yet infinitely long, a time ago—and we began rising and turning to test the wind for our 618 | crossing through the pass. At a very high level there must have been great disturbance, since 619 | the ice-dust clouds of the zenith were doing all sorts of fantastic things; but at 24,000 feet, the 620 | height we needed for the pass, we found navigation quite practicable. As we drew close to the 621 | jutting peaks the wind’s strange piping again became manifest, and I could see Danforth’s 622 | hands trembling at the controls. Rank amateur though I was, I thought at that moment that I 623 | might be a better navigator than he in effecting the dangerous crossing between pinnacles; and 624 | when I made motions to change seats and take over his duties he did not protest. I tried to keep 625 | all my skill and self-possession about me, and stared at the sector of reddish farther sky betwixt 626 | the walls of the pass—resolutely refusing to pay attention to the puffs of mountain-top vapour, 627 | and wishing that I had wax-stopped ears like Ulysses’ men off the Sirens’ coast to keep that 628 | disturbing wind-piping from my consciousness. 629 | But Danforth, released from his piloting and keyed up to a dangerous nervous pitch, could not 630 | keep quiet. I felt him turning and wriggling about as he looked back at the terrible receding city, 631 | ahead at the cave-riddled, cube-barnacled peaks, sidewise at the bleak sea of snowy, rampart- 632 | strown foothills, and upward at the seething, grotesquely clouded sky. It was then, just as I was 633 | trying to steer safely through the pass, that his mad shrieking brought us so close to disaster by 634 | shattering my tight hold on myself and causing me to fumble helplessly with the controls for a 635 | moment. A second afterward my resolution triumphed and we made the crossing safely—yet I 636 | am afraid that Danforth will never be the same again. 637 | I have said that Danforth refused to tell me what final horror made him scream out so 638 | insanely—a horror which, I feel sadly sure, is mainly responsible for his present breakdown. We 639 | had snatches of shouted conversation above the wind’s piping and the engine’s buzzing as we 640 | reached the safe side of the range and swooped slowly down toward the camp, but that had 641 | mostly to do with the pledges of secrecy we had made as we prepared to leave the nightmare 642 | city. Certain things, we had agreed, were not for people to know and discuss lightly—and I would 643 | not speak of them now but for the need of heading off that Starkweather-Moore Expedition, and 644 | others, at any cost. It is absolutely necessary, for the peace and safety of mankind, that some of 645 | earth’s dark, dead corners and unplumbed depths be let alone; lest sleeping abnormalities 646 | wake to resurgent life, and blasphemously surviving nightmares squirm and splash out of their 647 | black lairs to newer and wider conquests. 648 | 73 / 74 649 | Full Text Archive 650 | https://www.fulltextarchive.comAll that Danforth has ever hinted is that the final horror was a mirage. It was not, he declares, 651 | anything connected with the cubes and caves of echoing, vaporous, wormily honeycombed 652 | mountains of madness which we crossed; but a single fantastic, daemoniac glimpse, among the 653 | churning zenith-clouds, of what lay back of those other violet westward mountains which the Old 654 | Ones had shunned and feared. It is very probable that the thing was a sheer delusion born of 655 | the previous stresses we had passed through, and of the actual though unrecognised mirage of 656 | the dead transmontane city experienced near Lake’s camp the day before; but it was so real to 657 | Danforth that he suffers from it still. 658 | He has on rare occasions whispered disjointed and irresponsible things about “the black pit”, 659 | “the carven rim”, “the proto-shoggoths”, “the windowless solids with five dimensions”, “the 660 | nameless cylinder”, “the elder pharos”, “Yog-Sothoth”, “the primal white jelly”, “the colour out 661 | of space”, “the wings”, “the eyes in darkness”, “the moon-ladder”, “the original, the eternal, 662 | the undying”, and other bizarre conceptions; but when he is fully himself he repudiates all this 663 | and attributes it to his curious and macabre reading of earlier years. Danforth, indeed, is known 664 | to be among the few who have ever dared go completely through that worm-riddled copy of the 665 | Necronomicon kept under lock and key in the college library. 666 | The higher sky, as we crossed the range, was surely vaporous and disturbed enough; and 667 | although I did not see the zenith I can well imagine that its swirls of ice-dust may have taken 668 | strange forms. Imagination, knowing how vividly distant scenes can sometimes be reflected, 669 | refracted, and magnified by such layers of restless cloud, might easily have supplied the 670 | rest—and of course Danforth did not hint any of those specific horrors till after his memory had 671 | had a chance to draw on his bygone reading. He could never have seen so much in one 672 | instantaneous glance. 673 | At the time his shrieks were confined to the repetition of a single mad word of all too obvious 674 | source: 675 | “Tekeli-li! Tekeli-li!” 676 | Powered by TCPDF (www.tcpdf.org) 677 | 74 / 74 678 | 679 | Full Text Archive 680 | https://www.fulltextarchive.comAt the Mountains of Madness By H. P. Lovecraft 681 | I. 682 | I am forced into speech because men of science have refused to follow my advice without 683 | knowing why. It is altogether against my will that I tell my reasons for opposing this 684 | contemplated invasion of the antarctic—with its vast fossil-hunt and its wholesale boring and 685 | melting of the ancient ice-cap—and I am the more reluctant because my warning may be in vain. 686 | Doubt of the real facts, as I must reveal them, is inevitable; yet if I suppressed what will seem 687 | extravagant and incredible there would be nothing left. The hitherto withheld photographs, both 688 | ordinary and aërial, will count in my favour; for they are damnably vivid and graphic. Still, they 689 | will be doubted because of the great lengths to which clever fakery can be carried. The ink 690 | drawings, of course, will be jeered at as obvious impostures; notwithstanding a strangeness of 691 | technique which art experts ought to remark and puzzle over. 692 | In the end I must rely on the judgment and standing of the few scientific leaders who have, on 693 | the one hand, sufficient independence of thought to weigh my data on its own hideously 694 | convincing merits or in the light of certain primordial and highly baffling myth-cycles; and on the 695 | other hand, sufficient influence to deter the exploring world in general from any rash and 696 | overambitious programme in the region of those mountains of madness. It is an unfortunate fact 697 | that relatively obscure men like myself and my associates, connected only with a small 698 | university, have little chance of making an impression where matters of a wildly bizarre or highly 699 | controversial nature are concerned. 700 | 1 / 74 701 | Full Text Archive 702 | https://www.fulltextarchive.comIt is further against us that we are not, in the strictest sense, specialists in the fields which came 703 | primarily to be concerned. As a geologist my object in leading the Miskatonic University 704 | Expedition was wholly that of securing deep-level specimens of rock and soil from various parts 705 | of the antarctic continent, aided by the remarkable drill devised by Prof. Frank H. Pabodie of our 706 | engineering department. I had no wish to be a pioneer in any other field than this; but I did hope 707 | that the use of this new mechanical appliance at different points along previously explored paths 708 | would bring to light materials of a sort hitherto unreached by the ordinary methods of collection. 709 | Pabodie’s drilling apparatus, as the public already knows from our reports, was unique and 710 | radical in its lightness, portability, and capacity to combine the ordinary artesian drill principle 711 | with the principle of the small circular rock drill in such a way as to cope quickly with strata of 712 | varying hardness. Steel head, jointed rods, gasoline motor, collapsible wooden derrick, 713 | dynamiting paraphernalia, cording, rubbish-removal auger, and sectional piping for bores five 714 | inches wide and up to 1000 feet deep all formed, with needed accessories, no greater load than 715 | three seven-dog sledges could carry; this being made possible by the clever aluminum alloy of 716 | which most of the metal objects were fashioned. Four large Dornier aëroplanes, designed 717 | especially for the tremendous altitude flying necessary on the antarctic plateau and with added 718 | fuel-warming and quick-starting devices worked out by Pabodie, could transport our entire 719 | expedition from a base at the edge of the great ice barrier to various suitable inland points, and 720 | from these points a sufficient quota of dogs would serve us. 721 | We planned to cover as great an area as one antarctic season—or longer, if absolutely 722 | necessary—would permit, operating mostly in the mountain-ranges and on the plateau south of 723 | Ross Sea; regions explored in varying degree by Shackleton, Amundsen, Scott, and Byrd. With 724 | frequent changes of camp, made by aëroplane and involving distances great enough to be of 725 | geological significance, we expected to unearth a quite unprecedented amount of material; 726 | especially in the pre-Cambrian strata of which so narrow a range of antarctic specimens had 727 | previously been secured. We wished also to obtain as great as possible a variety of the upper 728 | fossiliferous rocks, since the primal life-history of this bleak realm of ice and death is of the 729 | highest importance to our knowledge of the earth’s past. That the antarctic continent was once 730 | temperate and even tropical, with a teeming vegetable and animal life of which the lichens, 731 | marine fauna, arachnida, and penguins of the northern edge are the only survivals, is a matter 732 | of common information; and we hoped to expand that information in variety, accuracy, and 733 | detail. When a simple boring revealed fossiliferous signs, we would enlarge the aperture by 734 | blasting in order to get specimens of suitable size and condition. 735 | Our borings, of varying depth according to the promise held out by the upper soil or rock, were 736 | to be confined to exposed or nearly exposed land surfaces—these inevitably being slopes and 737 | ridges because of the mile or two-mile thickness of solid ice overlying the lower levels. We could 738 | not afford to waste drilling depth on any considerable amount of mere glaciation, though 739 | Pabodie had worked out a plan for sinking copper electrodes in thick clusters of borings and 740 | melting off limited areas of ice with current from a gasoline-driven dynamo. It is this plan—which 741 | we could not put into effect except experimentally on an expedition such as ours—that the 742 | coming Starkweather-Moore Expedition proposes to follow despite the warnings I have issued 743 | since our return from the antarctic. 744 | 2 / 74 745 | 746 | Full Text Archive 747 | https://www.fulltextarchive.comTill A’ the Seas By R. H. Barlow and H. P. Lovecraft 748 | Upon an eroded cliff-top rested the man, gazing far across the valley. Lying thus, he could see a 749 | great distance, but in all the sere expanse there was no visible motion. Nothing stirred the dusty 750 | plain, the disintegrated sand of long-dry river-beds, where once coursed the gushing streams of 751 | Earth’s youth. There was little greenery in this ultimate world, this final stage of mankind’s 752 | prolonged presence upon the planet. For unnumbered aeons the drought and sandstorms had 753 | ravaged all the lands. The trees and bushes had given way to small, twisted shrubs that 754 | persisted long through their sturdiness; but these, in turn, perished before the onslaught of 755 | coarse grasses and stringy, tough vegetation of strange evolution. 756 | The ever-present heat, as Earth drew nearer to the sun, withered and killed with pitiless rays. It 757 | had not come at once; long aeons had gone before any could feel the change. And all through 758 | those first ages man’s adaptable form had followed the slow mutation and modelled itself to fit 759 | the more and more torrid air. Then the day had come when men could bear their hot cities but 760 | ill, and a gradual recession began, slow yet deliberate. Those towns and settlements closest to 761 | the equator had been first, of course, but later there were others. Man, softened and exhausted, 762 | could cope no longer with the ruthlessly mounting heat. It seared him as he was, and evolution 763 | was too slow to mould new resistances in him. 764 | Yet not at first were the great cities of the equator left to the spider and the scorpion. In the early 765 | years there were many who stayed on, devising curious shields and armours against the heat 766 | and the deadly dryness. These fearless souls, screening certain buildings against the 767 | encroaching sun, made miniature worlds of refuge wherein no protective armour was needed. 768 | They contrived marvellously ingenious things, so that for a while men persisted in the rusting 769 | towers, hoping thereby to cling to old lands till the searing should be over. For many would not 770 | believe what the astronomers said, and looked for a coming of the mild olden world again. But 771 | one day the men of Dath, from the new city of Niyara, made signals to Yuanario, their 772 | immemorially ancient capital, and gained no answer from the few who remained therein. And 773 | when explorers reached that millennial city of bridge-linked towers they found only silence. 774 | There was not even the horror of corruption, for the scavenger lizards had been swift. 775 | Only then did the people fully realize that these cities were lost to them; know that they must 776 | forever abandon them to nature. The other colonists in the hot lands fled from their brave posts, 777 | and total silence reigned within the high basalt walls of a thousand empty towns. Of the dense 778 | throngs and multitudinous activities of the past, nothing finally remained. There now loomed 779 | against the rainless deserts only the blistered towers of vacant houses, factories, and structures 780 | of every sort, reflecting the sun’s dazzling radiance and parching in the more and more 781 | intolerable heat. 782 | 1 / 8 783 | Full Text Archive 784 | https://www.fulltextarchive.comMany lands, however, had still escaped the scorching blight, so that the refugees were soon 785 | absorbed in the life of a newer world. During strangely prosperous centuries the hoary deserted 786 | cities of the equator grew half-forgotten and entwined with fantastic fables. Few thought of those 787 | spectral, rotting towers . . . those huddles of shabby walls and cactus-choked streets, darkly 788 | silent and abandoned. . . . 789 | Wars came, sinful and prolonged, but the times of peace were greater. Yet always the swollen 790 | sun increased its radiance as Earth drew closer to its fiery parent. It was as if the planet meant 791 | to return to that source whence it was snatched, aeons ago, through the accidents of cosmic 792 | growth. 793 | After a time the blight crept outward from the central belt. Southern Yarat burned as a 794 | tenantless desert—and then the north. In Perath and Baling, those ancient cities where brooding 795 | centuries dwelt, there moved only the scaly shapes of the serpent and the salamander, and at 796 | last Loton echoed only to the fitful falling of tottering spires and crumbling domes. 797 | Steady, universal, and inexorable was the great eviction of man from the realms he had always 798 | known. No land within the widening stricken belt was spared; no people left unrouted. It was an 799 | epic, a titan tragedy whose plot was unrevealed to the actors—this wholesale desertion of the 800 | cities of men. It took not years or even centuries, but millennia of ruthless change. And still it 801 | kept on—sullen, inevitable, savagely devastating. 802 | Agriculture was at a standstill, the world fast became too arid for crops. This was remedied by 803 | artificial substitutes, soon universally used. And as the old places that had known the great 804 | things of mortals were left, the loot salvaged by the fugitives grew smaller and smaller. Things of 805 | the greatest value and importance were left in dead museums—lost amid the centuries—and in 806 | the end the heritage of the immemorial past was abandoned. A degeneracy both physical and 807 | cultural set in with the insidious heat. For man had so long dwelt in comfort and security that this 808 | exodus from past scenes was difficult. Nor were these events received phlegmatically; their very 809 | slowness was terrifying. Degradation and debauchery were soon common; government was 810 | disorganized, and the civilizations aimlessly slid back toward barbarism. 811 | When, forty-nine centuries after the blight from the equatorial belt, the whole western 812 | hemisphere was left unpeopled, chaos was complete. There was no trace of order or decency in 813 | the last scenes of this titanic, wildly impressive migration. Madness and frenzy stalked through 814 | them, and fanatics screamed of an Armageddon close at hand. 815 | 2 / 8 816 | 817 | R 291854Z JAN 21 MID600050190282U 818 | FM CNO WASHINGTON DC 819 | TO NAVADMIN 820 | INFO CNO WASHINGTON DC 821 | BT 822 | UNCLAS 823 | 824 | NAVADMIN 026/21 825 | 826 | PASS TO OFFICE CODES: 827 | FM CNO WASHINGTON DC//N1// 828 | INFO CNO WASHINGTON DC//N1// 829 | MSGID/GENADMIN/CNO WASHINGTON DC/N1/JAN// 830 | 831 | SUBJ/DEPARTMENT OF DEFENSE COVID-19 TESTING PRIOR TO OVERSEAS TRAVEL (UPDATE 832 | 1)// 833 | 834 | REF/A/NAVADMIN/OPNAV/072031ZJAN21// 835 | REF/B/EXECUTIVE ORDER/13998/21JAN21// 836 | REF/C/USD P&R MEMO/29DEC20// 837 | REF/D/CDC ORDER/12JAN21// 838 | REF/E/GENADMIN/USTRANSCOM/230226ZJAN21// 839 | REF/F/NAVADMIN/OPNAV/042056ZNOV20// 840 | 841 | NARR/ REF A IS NAVADMIN 03/21, DEPARTMENT OF DEFENSE COVID-19 842 | TESTING PRIOR TO OVERSEAS TRAVEL. 843 | REF B IS EXECUTIVE ORDER 13998, PROMOTING COVID-19 SAFETY IN DOMESTIC AND 844 | INTERNATIONAL TRAVEL. 845 | REF C IS FORCE HEALTH PROTECTION GUIDANCE (SUPPLEMENT 14). 846 | REF D IS CENTERS FOR DISEASE CONTROL AND PREVENTION ORDER REQUIREMENT FOR 847 | NEGATIVE PRE-DEPARTURE COVID-19 TEST RESULT OR DOCUMENTATION OF RECOVERY FROM 848 | COVID-19 FOR ALL AIRLINE OR OTHER AIRCRAFT PASSENGERS ARRIVING INTO THE 849 | UNITED STATES FROM ANY FOREIGN COUNTRY. 850 | REF E IS USTRANSCOM AMENDED GUIDANCE TO THE JOINT FORCE UPDATING COVID-19 851 | TRAVEL REQUIREMENTS FOR TRAVELERS MOVING FROM OCONUS TO CONUS DESTINATIONS IN 852 | RESPONSE TO FORCE HEALTH PROTECTION SUPPLEMENT 14. 853 | REF F IS NAVADMIN 298/20, US NAVY COVID-19 STANDARDIZED OPERATIONAL GUIDANCE 854 | VERSION 3.1. 855 | 856 | RMKS/1. This NAVADMIN adds requirements for COVID-19 testing prior to 857 | international travel, in line with reference (a), to include testing prior to 858 | entry into the U.S. from a foreign country, as described in references (b) 859 | through (e). Reference (a) is cancelled. 860 | 861 | 2. The dynamic nature of the COVID-19 pandemic requires the implementation 862 | of measures to mitigate the risk of COVID-19 spread. 863 | Many countries, to include the U.S., now require negative test results before 864 | arrival. In line with reference (c), and to ensure Service Members and 865 | families are not refused travel or delayed, all travelers executing 866 | international orders, to include returning to the U.S. from a foreign 867 | country, are required to have a negative viral COVID-19 test (molecular or 868 | antigen). An antigen test may be used for testing prior to travel when a 869 | molecular test (such as polymerase chain reaction (PCR) or Abbott ID NOW) is 870 | not available. 871 | However, a molecular test is the preferred test prior to travel. If the 872 | destination location requires a specific test, test timing, or test result 873 | format, travelers must follow the stricter requirement. 874 | 3. Service Members and/or family members must do the following 875 | a. Possess, and present on request, proof of a negative viral COVID-19 876 | test administered within 72 hours of embarkation for international travel for 877 | each traveling family member. Valid proof may be electronic or printed for 878 | commercial flights. Printed proof will be requested prior to Air Mobility 879 | Command (AMC) Patriot Express (PE) (i.e., rotator) or Military aircraft 880 | flights. 881 | Travelers should verify specific country entry requirements to avoid delays, 882 | fines, or other complications upon arrival. 883 | Note: If previously infected with COVID-19, a molecular COVID-19 test may 884 | still produce a positive result. If unable to achieve a negative result, 885 | consult with a medical provider, the Foreign Clearance Guide (FCG), and/or 886 | your travel provider to determine the necessary medical clearance 887 | documentation. Documents likely include the proof of COVID-19 positive 888 | results, written documentation from the treating physician that the traveler 889 | has recovered, and the date of the onset of symptoms or positive test. 890 | b. Review the FCG (https://www.fcg.pentagon.mil/Fcg.cfm) to determine 891 | the exact entry requirements for the final destination (does not apply to 892 | routine intermediate stops). COVID-19 travel information for travelers 893 | returning to the United States may be found on the Department of State 894 | website (https://travel.state.gov/content/travel/ 895 | en/traveladvisories/ea/covid-19-information.html). Additionally, travelers 896 | are encouraged to register for the Smart Traveler Enrollment Program 897 | (https://step.state.gov) for a free service enabling travelers to enroll 898 | their trip with the nearest U.S. Embassy or Consulate. This service provides 899 | updates from the embassy regarding travel requirements and safety conditions 900 | in the destination country. 901 | c. Review associated destination requirements, such as type of test, 902 | timing or test result format, to determine specifics for entry that may be 903 | more restrictive than service policy, e.g., testing within 72 hours from 904 | arrival instead of embarkation - in this case, travelers must execute their 905 | testing in order to arrive at their embarkation site with proof and have 906 | sufficient time remaining to travel to their destination prior to the 907 | requirement expiring. 908 | d. Plan intermediate travel and leave. Do not place you and your family 909 | in extremis should an unexpected restriction of movement (ROM) be required 910 | while traveling prior to embarkation. During trip planning, ensure you 911 | identify facilities along your route of travel that will accommodate you or a 912 | family member who may become COVID-19 positive. Additionally, financially 913 | plan for an extended trip and/or additional tests should they become 914 | necessary. Service Members are encouraged to use their Government Travel 915 | Charge Card (GTCC) for travel expenses on official orders as this will assist 916 | in alleviating personal financial impact. TRICARE covers COVID-19 tests when 917 | medically necessary and appropriate. Visit: 918 | https://www.tricare.mil/covid19testing for more details on COVID-19 testing 919 | coverage. TRICARE does not fund strictly travel-based tests. If not using a 920 | military facility, Service Members traveling on official orders should be 921 | prepared to pay for tests and be refunded via travel claim, in line with the 922 | Joint Travel Regulations. For those that choose not to use the GTCC or do 923 | not possess a GTCC, advanced travel pay is encouraged since molecular-based 924 | tests may be expensive. If paying for a commercial test that requires an 925 | upfront fee (copay), Service Members should ensure to submit the full testing 926 | cost via travel claim. 927 | e. Take responsibility for testing. The service is providing the 928 | following options to assist with individual travel situations, but it is the 929 | responsibility of the traveler to ensure all requirements have been met: 930 | (1) When traveling internationally from the U.S.: 931 | (a) Military treatment facilities (MTF) will execute free tests 932 | for Service Members and dependents in possession of international orders in 933 | line with reference (c). Tests should be scheduled prior to arrival at the 934 | MTF. MTFs should deliver hard copy documentation of test results within 24 935 | hours of testing to the Service Members and dependents. This allows for 936 | testing in a different area than former duty stations to account for leave or 937 | training in route. A non-inclusive list of preferred MTFs may be found on 938 | the Overseas PCS COVID-19 Testing Fact Sheet via the MyNavy Portal 939 | (https://www.mnp.navy.mil/covid-19.html). 940 | Note: Most MTFs are closed during the weekend and holidays. 941 | Travelers are encouraged to plan for alternate testing solutions if boarding 942 | a flight at the beginning of the week. 943 | (b) Travelers assigned to a U.S. Transportation Command 944 | (USTRANSCOM)/AMC PE flight (i.e., rotator) may be tested at the Madigan Army 945 | Medical Center (Seattle) or Walter Reed National Military Medical Center 946 | (Bethesda) if travel timing precludes testing prior to arriving at the PE 947 | terminal. If using this option, travelers should appear in person 48 hours 948 | prior to boarding at their respective Aerial Port of Embarkation (APOE). 949 | Walk-up testing is located in temporary shelters outside the facilities. 950 | Direct communication with the testing sites is limited. Applicable contact 951 | information as follows: 952 | i. AMC Gateway, Baltimore, MD: 953 | Phone - (609) 253-8825 954 | Schedule - Limited operations 955 | ii. Walter Reed National Military Medical Center, 956 | Bethesda, MD: 957 | Phone (Central Screening) (571) 335-9985 958 | Schedule - 0700-1600 EST, no weekends or 959 | holidays 960 | iii. AMC Gateway, Seattle, WA: 961 | Phone - (253) 982-3504/0555 962 | Schedule - Limited operations. 963 | iv. Madigan Army Medical Center, McChord, WA: 964 | Phone (PrevMed) (253) 968-4443 965 | Schedule - 0600-1700 PST, weekends/no holidays 966 | (c) Commercial sites that support antigen or molecular-based 967 | tests (e.g., PCR and Abbott ID NOW) may be used to satisfy the requirement. 968 | See planning and cost considerations above. 969 | (2) When traveling to the U.S. from a foreign country: 970 | (a) Travelers returning to the U.S. should primarily use their 971 | closest U.S. installation MTF. COVID-19 testing results will be available 972 | via TRICARE Online, often within 24 hours of testing. 973 | Results may be printed, if hard copies are required. A list of preferred 974 | international MTFs may be found on the Overseas PCS COVID-19 Testing Fact 975 | Sheet via the MyNavy Portal (https://www.mnp.navy.mil/covid-19.html). 976 | (b) International commercial sites that support antigen or 977 | molecular-based tests may also be used to satisfy the requirement. However, 978 | travelers should consult with their local installation, military liaison, or 979 | U.S. Embassy prior to doing so. See planning and cost considerations above. 980 | f. Travelers that test positive for COVID-19 while executing 981 | international travel must do the following: 982 | (1) Immediately execute ROM procedures. If a member of a family unit 983 | tests positive, the entire family unit must commence ROM. Travelers may not 984 | proceed separately. If not already in place at suitable lodging, priority 985 | for ROM site should be given to the following in order: 1) APOE, 2) home or 986 | detaching installation, 3) nearby family member, 4) nearby acceptable 987 | military installation and 5) nearby acceptable hotel. Call ahead to the 988 | preferred ROM site for notification and support. If no ROM site is available 989 | and additional support is required, call the MyNavy Career Center (MNCC) 990 | contact information below. If at a U.S. PE APOE, USTRANSCOM/AMC will take 991 | immediate responsibility for the travelers, to include medical, lodging, pet 992 | services, meals and transportation support. 993 | AMC will direct travelers to nearby military installations or participating 994 | lodging sites. While in ROM, travelers must strictly adhere to force 995 | protection health guidance and return to work criteria, in line with 996 | reference (f). Do not proceed until consulting with your detailer and/or AMC 997 | (if at a U.S. PE APOE). 998 | (2) Contact the MNCC for initial notification and additional support. 999 | Available 24/7, by phone at 1-833-330-MNCC (6622) or email at 1000 | askmncc(at)navy.mil. Provide valid contact information and proof of positive 1001 | test result(s) to the servicing MNCC agent. The servicing MNCC agent will 1002 | take appropriate action to notify the Service Member detailer, Navy Passenger 1003 | Transportation Office (NAVPTO), and the OPNAV COVID-19 Taskforce Liaison. 1004 | (3) Continue communicating with your detailer and NAVPTO agent for 1005 | orders modification (ORDMOD) and logistics support. Upon notification from 1006 | the MNCC, the detailer and NAVPTO agent will contact the Service Member to 1007 | discuss an ORDMOD and re-book future travel. ORDMODs will ensure per diem is 1008 | available and reimbursable for qualified expenses incurred during the ROM 1009 | period. Service Members may use the GTCC if issued. Additionally, Service 1010 | Members may request a travel advance through the MNCC regardless of GTCC 1011 | possession. 1012 | (4) Secure from ROM. Once the travelers meet return to work 1013 | criteria, in line with reference (f) and in conjunction with any medical 1014 | provider guidance, they may continue execution of their orders after 1015 | conferring with detailers and/or AMC (if at a U.S. PE APOE). Service Members 1016 | and detailers should be proactive to continue travel as soon as possible. 1017 | 1018 | 4. Command Pay and Personnel Administrators must do the following before a 1019 | Service Member detaches: 1020 | a. Notify Service Member of the testing requirement for international 1021 | travel and options. Keep a record of the plans of the Service Member for 1022 | testing, to include the intended facility site and date. 1023 | b. Counsel Service Members on the expense for commercial testing and 1024 | discuss options for the potential financial burden, e.g. GTCC, advanced 1025 | travel pay, reimbursement, etc. 1026 | c. Ensure the Service Member has detaching command contact information 1027 | in case additional support is needed during an intermediate ROM status. 1028 | 1029 | 5. Travelers with valid proof of COVID-19 vaccination are currently not 1030 | exempt from the testing requirement. 1031 | 1032 | 6. Contact the MNCC with any questions, issues or to provide feedback at 1033 | 833-330-MNCC/6622 or via e-mail at askmncc(at)navy.mil. 1034 | 1035 | 7. Released by Vice Admiral John B. Nowell, Jr, N1.// 1036 | 1037 | BT 1038 | #0001 1039 | NNNN 1040 | UNCLASSIFIED// 1041 | 1042 | TESTING IMPLEMENTATIONS OF DES 1043 | ------------------------------ 1044 | 1045 | Ronald L. Rivest 1046 | MIT Laboratory for Computer Science 1047 | Cambridge, Mass. 02139 1048 | 2/23/85 1049 | 1050 | 1051 | ABSTRACT 1052 | -------- 1053 | 1054 | We present a simple way to test the correctness of a DES implementation: 1055 | Use the recurrence relation: 1056 | 1057 | X0 = 9474B8E8C73BCA7D (hexadecimal) 1058 | 1059 | X(i+1) = IF (i is even) THEN E(Xi,Xi) ELSE D(Xi,Xi) 1060 | 1061 | to compute a sequence of 64-bit values: X0, X1, X2, ..., X16. Here 1062 | E(X,K) denotes the DES encryption of X using key K, and D(X,K) denotes 1063 | the DES decryption of X using key K. If you obtain 1064 | 1065 | X16 = 1B1A2DDB4C642438 1066 | 1067 | your implementation does not have any of the 36,568 possible single-fault 1068 | errors described herein. 1069 | 1070 | 1071 | INTRODUCTION 1072 | ------------ 1073 | 1074 | The Data Encryption Standard (DES) has been approved by a variety of 1075 | organizations (e.g. the U.S. government) for use in cryptographic applications. 1076 | The DES algorithm was published by the National Bureau of Standards [FIPS46]. 1077 | 1078 | The National Bureau of Standards has an ongoing program to validate the 1079 | correctness of hardware DES implementations. Their testing procedure, 1080 | described in [Ga80a], consists of 291 fixed test cases followed by twelve 1081 | million randomly chosen test cases. A hardware implementation of DES that 1082 | passes this test is awarded a "Validation Certficate". 1083 | 1084 | The above testing procedure provides good evidence that the DES hardware was 1085 | correctly designed and functioning properly at the time of validation. 1086 | However, the user of a DES implementation would like to maintain the currency 1087 | of this validation -- this would be "maintenance testing" in contrast 1088 | to the "validation" service provided by NBS. The purpose of maintenance 1089 | testing is to ensure that the DES implementation is still correct, i.e. 1090 | that no operational fault has appeared since the last time the the device 1091 | was tested. Also note that NBS's validation procedure only inspects single 1092 | instances of a hardware device; the user of a DES device wants to ensure that 1093 | his device is correctly functioning (even though the design is OK, the device 1094 | may have developed faults). 1095 | 1096 | One way to perform maintenance testing is to use two or more separate 1097 | implementations and compare their results after each operation. If they 1098 | ever differ, then one of the units has ceased to operate correctly. The 1099 | difficulty with this approach is the cost of providing the redundant 1100 | implementation. 1101 | 1102 | If a redundant implementation is infeasible, then the implementation may 1103 | be testing by comparing its input-output behaviour with that of a correct 1104 | implementation. This paper provides such a procedure. 1105 | 1106 | It is desired to have a relatively short test procedure, since the 1107 | maintenance testing may be taking place in parallel with ordinary operations, 1108 | and the cost of the test procedure may affect the unit's overall 1109 | efficiency. The test specified in this paper only requires 16 DES operations. 1110 | 1111 | It is desirable to have an effective test procedure, in the 1112 | sense that implementation errors are very likely to be caught. The test 1113 | specified here will catch any of the 36,568 single-fault implementation 1114 | errors specified later on. 1115 | 1116 | We next specify the DES implementation and error model, and then describe 1117 | the experimental results that yielded the proposed maintenance test. 1118 | 1119 | 1120 | DES IMPLEMENTATION 1121 | ------------------ 1122 | 1123 | Our DES implementation follows the specifications for DES: 1124 | 1125 | DES(X,K,E) 1126 | -- X is a 64-bit input value 1127 | -- K is a 64-bit key 1128 | -- E is TRUE if we are encrypting, else FALSE 1129 | (1) Apply permutation IP to X, resulting in value LR (64 bits). 1130 | (2) Apply selector PC1 to K, resulting in value CD (56 bits). 1131 | (3) For ROUND = 1, 2, ..., 16: 1132 | (3a) if E is TRUE then 1133 | if SHIFTS[ROUND] = 1 1134 | then apply permutation LSH1 to CD (in place). 1135 | else apply permutation LSH2 to CD (in place). 1136 | (3b) Apply selector PC2 to CD resulting in value KI (48 bits) 1137 | (3c) If E is FALSE then 1138 | if SHIFTS[17-ROUND] = 1 1139 | then apply permutation RSH1 to CD (in place). 1140 | else apply permutation RSH2 to CD (in place). 1141 | (3d) Apply selector E to LR, resulting in value RE (48 bits). 1142 | (3e) XOR RE with KI, resulting in value RX (48 bits). 1143 | (3f) Break RX into 8 6-bit blocks, and replace the i-th block 1144 | Yi with the result of looking up Yi in S-box i. 1145 | Concatenate these 4-bit results together to get the 32-bit 1146 | value SOUT. 1147 | (3g) Apply permutation P to SOUT resulting in value FOUT (32 1148 | bits). 1149 | (3h) Replace the left half of LR with the XOR of FOUT and the 1150 | left half of LR. 1151 | (3i) If ROUND is not 16, apply permutation SWAP to LR (in 1152 | place). 1153 | (4) Apply permutation IPINV to LR resulting in value OUT (64 bits). 1154 | Return OUT as the result of this DES operation. 1155 | 1156 | 1157 | This implementation makes uses of the following tables and functions (for 1158 | the details of these tables see [FIPS46]): 1159 | 1160 | (1) Permutations and selectors: IP, PC1, PC2, LSH1, LSH2, RSH1, RSH2, 1161 | E, P, SWAP, IPINV. 1162 | Each permutation or selector Q has a length LENGTH(Q) and a range 1163 | 1...RANGE(Q). For permutations LENGTH(Q)=RANGE(Q). As an example, 1164 | the selector PC2 = [14, 17, ..., 32] has length 48 and range 56, 1165 | while permutation SWAP = [33, 34, ..., 32] has length 64 and range 1166 | 64. 1167 | 1168 | (2) The table SHIFTS = [1, 1, 2, ..., 1] of length 16 specifying the 1169 | number of left shifts of the key registers to perform during 1170 | each round. 1171 | 1172 | (3) The XOR (exclusive-OR) functions used in steps (3e) and (3h), on 1173 | input vectors of lengths 48 and 32, respectively. 1174 | 1175 | (4) The S-box tables used in step (3f). 1176 | 1177 | 1178 | 1179 | ERROR MODEL 1180 | ----------- 1181 | 1182 | We now describe our model of implementation errors. We consider the following 1183 | set of single-fault errors: 1184 | 1185 | (1) "Wiring errors": For each permutation or selector Q, we consider 1186 | the possibility that each of the output elements might 1187 | -- be stuck at 0 1188 | -- be stuck at 1 1189 | -- incorrectly specify some other value in 1...RANGE(Q). 1190 | There are thus LENGTH(Q)*(RANGE(Q)+1) possible errors for Q. 1191 | 1192 | Permutation Number 1193 | or of Possible 1194 | Selector LENGTH RANGE Errors 1195 | ------------ ------- ------ ----------- 1196 | IP 64 64 4160 1197 | PC1 56 64 3640 1198 | PC2 48 56 2736 1199 | LSH1 56 56 3192 1200 | LSH2 56 56 3192 1201 | RSH1 56 56 3192 1202 | RHS2 56 56 3192 1203 | E 48 32 1584 1204 | P 32 32 1056 1205 | SWAP 64 64 4160 1206 | IPINV 64 64 4160 1207 | ----------------------------------------------------------- 1208 | Total Number of Possible Wiring Errors........34,264 1209 | 1210 | (2) Shift Errors: For each of the 16 positions of the SHIFTS table, 1211 | one error possibility is that the table entry could be incorrectly 1212 | specified (i.e. specify a shift of 1 when a shift of two was 1213 | desired, or vice versa). 1214 | 1215 | Total Number of Possible Shifting Errors..........16 1216 | 1217 | (3) XOR errors: For each XOR gate (there are 48 used in step (3e) 1218 | and 32 used in step (3h)) the following errors are possible: 1219 | -- stuck at 0 1220 | -- stuck at 1 1221 | -- producing the negation of the desired result. 1222 | 1223 | Total Number of Possible XOR Errors..............240 1224 | 1225 | (4) SBOX errors: For each of the 8 * 64 * 4 = 2048 bits of the S-box 1226 | tables, we consider the possibility that it might be incorrectly 1227 | stored (i.e. a 0 is stored instead of a 1, or vice versa). 1228 | 1229 | Total Number of Possible SBOX Errors............2048 1230 | 1231 | ------------------------------------------------------------------- 1232 | Total Number of Error Possibilities Considered........36,568 1233 | 1234 | We note that since our implementation is iterative, an error possibility 1235 | that can affect one round of encryption can affect every round. 1236 | 1237 | 1238 | DESIGN OF VALIDATION TEST 1239 | ------------------------- 1240 | 1241 | A simple iterative test was desired, so that the only storage 1242 | requirements were the desired starting value, the desired ending value, and 1243 | the number of DES iterations to perform. Alternation of encryption and 1244 | decryption was chosen in order to exercise both modes equally. The initial 1245 | value, X0 = 9474B8E8C73BCA7D (hexadecimal) was chosen with a short search, 1246 | in order to find a sequence that tested the S-box entries efficiently. 1247 | To generate a pseudo-random sequence of test values, the output of 1248 | one stage is used as both the key and data inputs for the next stage. 1249 | That is, the pseudo-random sequence is defined by: 1250 | 1251 | X0 = 9474B8E8C73BCA7D 1252 | 1253 | X(i+1) = E(Xi,Xi) if i is even 1254 | X(i+1) = D(Xi,Xi) if i is odd 1255 | 1256 | where E(X,K) denotes the encryption of the 64-bit value X with 64-bit key 1257 | K, and D(X,K) denotes the corresponding decryption. Note that the key is 1258 | a 64-bit value as specified in [FIPS46]; the 8 "parity" bits are discarded by 1259 | selector PC1. 1260 | 1261 | DES was implemented in software on an IBM PC. The correctness of this 1262 | implementation was checked using the 291 fixed tests given in [Ga77]. 1263 | The various error possibilities were implemented, and an experiment was 1264 | run to determine the least i such that comparing Xi to its expected value 1265 | detected all the proposed single-fault errors. Note that only one comparison 1266 | is needed; it is not necessary to compare the intermediate results with their 1267 | expected values. The following table was obtained: 1268 | 1269 | i Xi Number of errors NOT detected 1270 | -- ---------------- ----------------------------- 1271 | 0 9474B8E8C73BCA7D 36,568 1272 | 1 8DA744E0C94E5E17 14,170 1273 | 2 0CDB25E3BA3C6D79 4,842 1274 | 3 4784C4BA5006081F 2,866 1275 | 4 1CF1FC126F2EF842 1,550 1276 | 5 E4BE250042098D13 996 1277 | 6 7BFC5DC6ADB5797C 652 1278 | 7 1AB3B4D82082FB28 458 1279 | 8 C1576A14DE707097 274 1280 | 9 739B68CD2E26782A 180 1281 | 10 2A59F0C464506EDB 126 1282 | 11 A5C39D4251F0A81E 94 1283 | 12 7239AC9A6107DDB1 72 1284 | 13 070CAC8590241233 52 1285 | 14 78F87B6E3DFECF61 20 1286 | 15 95EC2578C2C433F0 4 1287 | 16 1B1A2DDB4C642438 0 1288 | 1289 | We conclude that at most 16 DES operations suffice to test the correctness 1290 | of a DES implementation, under our error model. 1291 | 1292 | We note that the number of rounds required was determined by the S-box tests. 1293 | The other error types were all detected within at most 11 iterations. 1294 | The number of DES operations required here is 3 less than in the set of S-box 1295 | tests published by NBS[Ga77]; a short search discovered the given starting 1296 | value. We leave it as an open problem to devise a test which uses 1297 | substantially fewer DES operations. 1298 | 1299 | We note that the above test may not catch multiple faults, or faults other than 1300 | those described in this paper. 1301 | 1302 | 1303 | ACKNOWLEDGEMENTS 1304 | ---------------- 1305 | 1306 | This work was supported by RSA Data Security, Inc. 1307 | 1308 | 1309 | REFERENCES 1310 | ---------- 1311 | 1312 | [FIPS46] "Specifications for the Data Encryption Standard." Federal 1313 | Information Processing Standards Publication 46 (January 15, 1977). 1314 | 1315 | [Ga80a] Gait, Jason. "Validating the Correctness of Hardware Implementations 1316 | of the NBS Data Encryption Standard". NBS Special Publication 500-20. 1317 | (Rev. September 1980). 1318 | 1319 | [Ga80b] Gait, Jason. "Maintenance Testing for the Data Encryption Standard." 1320 | NBS Special Publication 500-61. (August 1980). 1321 | 1322 | UTF-8 decoder capability and stress test 1323 | ---------------------------------------- 1324 | 1325 | Markus Kuhn - 2015-08-28 - CC BY 4.0 1326 | 1327 | This test file can help you examine, how your UTF-8 decoder handles 1328 | various types of correct, malformed, or otherwise interesting UTF-8 1329 | sequences. This file is not meant to be a conformance test. It does 1330 | not prescribe any particular outcome. Therefore, there is no way to 1331 | "pass" or "fail" this test file, even though the text does suggest a 1332 | preferable decoder behaviour at some places. Its aim is, instead, to 1333 | help you think about, and test, the behaviour of your UTF-8 decoder on a 1334 | systematic collection of unusual inputs. Experience so far suggests 1335 | that most first-time authors of UTF-8 decoders find at least one 1336 | serious problem in their decoder using this file. 1337 | 1338 | The test lines below cover boundary conditions, malformed UTF-8 1339 | sequences, as well as correctly encoded UTF-8 sequences of Unicode code 1340 | points that should never occur in a correct UTF-8 file. 1341 | 1342 | According to ISO 10646-1:2000, sections D.7 and 2.3c, a device 1343 | receiving UTF-8 shall interpret a "malformed sequence in the same way 1344 | that it interprets a character that is outside the adopted subset" and 1345 | "characters that are not within the adopted subset shall be indicated 1346 | to the user" by a receiving device. One commonly used approach in 1347 | UTF-8 decoders is to replace any malformed UTF-8 sequence by a 1348 | replacement character (U+FFFD), which looks a bit like an inverted 1349 | question mark, or a similar symbol. It might be a good idea to 1350 | visually distinguish a malformed UTF-8 sequence from a correctly 1351 | encoded Unicode character that is just not available in the current 1352 | font but otherwise fully legal, even though ISO 10646-1 doesn't 1353 | mandate this. In any case, just ignoring malformed sequences or 1354 | unavailable characters does not conform to ISO 10646, will make 1355 | debugging more difficult, and can lead to user confusion. 1356 | 1357 | Please check, whether a malformed UTF-8 sequence is (1) represented at 1358 | all, (2) represented by exactly one single replacement character (or 1359 | equivalent signal), and (3) the following quotation mark after an 1360 | illegal UTF-8 sequence is correctly displayed, i.e. proper 1361 | resynchronization takes place immediately after any malformed 1362 | sequence. This file says "THE END" in the last line, so if you don't 1363 | see that, your decoder crashed somehow before, which should always be 1364 | cause for concern. 1365 | 1366 | All lines in this file are exactly 79 characters long (plus the line 1367 | feed). In addition, all lines end with "|", except for the two test 1368 | lines 2.1.1 and 2.2.1, which contain non-printable ASCII controls 1369 | U+0000 and U+007F. If you display this file with a fixed-width font, 1370 | these "|" characters should all line up in column 79 (right margin). 1371 | This allows you to test quickly, whether your UTF-8 decoder finds the 1372 | correct number of characters in every line, that is whether each 1373 | malformed sequences is replaced by a single replacement character. 1374 | 1375 | Note that, as an alternative to the notion of malformed sequence used 1376 | here, it is also a perfectly acceptable (and in some situations even 1377 | preferable) solution to represent each individual byte of a malformed 1378 | sequence with a replacement character. If you follow this strategy in 1379 | your decoder, then please ignore the "|" column. 1380 | 1381 | 1382 | Here come the tests: | 1383 | | 1384 | 1 Some correct UTF-8 text | 1385 | | 1386 | You should see the Greek word 'kosme': "κόσμε" | 1387 | | 1388 | 2 Boundary condition test cases | 1389 | | 1390 | 2.1 First possible sequence of a certain length | 1391 | | 1392 | 2.1.1 1 byte (U-00000000): "" 1393 | 2.1.2 2 bytes (U-00000080): "€" | 1394 | 2.1.3 3 bytes (U-00000800): "ࠀ" | 1395 | 2.1.4 4 bytes (U-00010000): "𐀀" | 1396 | 2.1.5 5 bytes (U-00200000): "�����" | 1397 | 2.1.6 6 bytes (U-04000000): "������" | 1398 | | 1399 | 2.2 Last possible sequence of a certain length | 1400 | | 1401 | 2.2.1 1 byte (U-0000007F): "" 1402 | 2.2.2 2 bytes (U-000007FF): "߿" | 1403 | 2.2.3 3 bytes (U-0000FFFF): "￿" | 1404 | 2.2.4 4 bytes (U-001FFFFF): "����" | 1405 | 2.2.5 5 bytes (U-03FFFFFF): "�����" | 1406 | 2.2.6 6 bytes (U-7FFFFFFF): "������" | 1407 | | 1408 | 2.3 Other boundary conditions | 1409 | | 1410 | 2.3.1 U-0000D7FF = ed 9f bf = "퟿" | 1411 | 2.3.2 U-0000E000 = ee 80 80 = "" | 1412 | 2.3.3 U-0000FFFD = ef bf bd = "�" | 1413 | 2.3.4 U-0010FFFF = f4 8f bf bf = "􏿿" | 1414 | 2.3.5 U-00110000 = f4 90 80 80 = "����" | 1415 | | 1416 | 3 Malformed sequences | 1417 | | 1418 | 3.1 Unexpected continuation bytes | 1419 | | 1420 | Each unexpected continuation byte should be separately signalled as a | 1421 | malformed sequence of its own. | 1422 | | 1423 | 3.1.1 First continuation byte 0x80: "�" | 1424 | 3.1.2 Last continuation byte 0xbf: "�" | 1425 | | 1426 | 3.1.3 2 continuation bytes: "��" | 1427 | 3.1.4 3 continuation bytes: "���" | 1428 | 3.1.5 4 continuation bytes: "����" | 1429 | 3.1.6 5 continuation bytes: "�����" | 1430 | 3.1.7 6 continuation bytes: "������" | 1431 | 3.1.8 7 continuation bytes: "�������" | 1432 | | 1433 | 3.1.9 Sequence of all 64 possible continuation bytes (0x80-0xbf): | 1434 | | 1435 | "���������������� | 1436 | ���������������� | 1437 | ���������������� | 1438 | ����������������" | 1439 | | 1440 | 3.2 Lonely start characters | 1441 | | 1442 | 3.2.1 All 32 first bytes of 2-byte sequences (0xc0-0xdf), | 1443 | each followed by a space character: | 1444 | | 1445 | "� � � � � � � � � � � � � � � � | 1446 | � � � � � � � � � � � � � � � � " | 1447 | | 1448 | 3.2.2 All 16 first bytes of 3-byte sequences (0xe0-0xef), | 1449 | each followed by a space character: | 1450 | | 1451 | "� � � � � � � � � � � � � � � � " | 1452 | | 1453 | 3.2.3 All 8 first bytes of 4-byte sequences (0xf0-0xf7), | 1454 | each followed by a space character: | 1455 | | 1456 | "� � � � � � � � " | 1457 | | 1458 | 3.2.4 All 4 first bytes of 5-byte sequences (0xf8-0xfb), | 1459 | each followed by a space character: | 1460 | | 1461 | "� � � � " | 1462 | | 1463 | 3.2.5 All 2 first bytes of 6-byte sequences (0xfc-0xfd), | 1464 | each followed by a space character: | 1465 | | 1466 | "� � " | 1467 | | 1468 | 3.3 Sequences with last continuation byte missing | 1469 | | 1470 | All bytes of an incomplete sequence should be signalled as a single | 1471 | malformed sequence, i.e., you should see only a single replacement | 1472 | character in each of the next 10 tests. (Characters as in section 2) | 1473 | | 1474 | 3.3.1 2-byte sequence with last byte missing (U+0000): "�" | 1475 | 3.3.2 3-byte sequence with last byte missing (U+0000): "��" | 1476 | 3.3.3 4-byte sequence with last byte missing (U+0000): "���" | 1477 | 3.3.4 5-byte sequence with last byte missing (U+0000): "����" | 1478 | 3.3.5 6-byte sequence with last byte missing (U+0000): "�����" | 1479 | 3.3.6 2-byte sequence with last byte missing (U-000007FF): "�" | 1480 | 3.3.7 3-byte sequence with last byte missing (U-0000FFFF): "�" | 1481 | 3.3.8 4-byte sequence with last byte missing (U-001FFFFF): "���" | 1482 | 3.3.9 5-byte sequence with last byte missing (U-03FFFFFF): "����" | 1483 | 3.3.10 6-byte sequence with last byte missing (U-7FFFFFFF): "�����" | 1484 | | 1485 | 3.4 Concatenation of incomplete sequences | 1486 | | 1487 | All the 10 sequences of 3.3 concatenated, you should see 10 malformed | 1488 | sequences being signalled: | 1489 | | 1490 | "�����������������������������" | 1491 | | 1492 | 3.5 Impossible bytes | 1493 | | 1494 | The following two bytes cannot appear in a correct UTF-8 string | 1495 | | 1496 | 3.5.1 fe = "�" | 1497 | 3.5.2 ff = "�" | 1498 | 3.5.3 fe fe ff ff = "����" | 1499 | | 1500 | 4 Overlong sequences | 1501 | | 1502 | The following sequences are not malformed according to the letter of | 1503 | the Unicode 2.0 standard. However, they are longer then necessary and | 1504 | a correct UTF-8 encoder is not allowed to produce them. A "safe UTF-8 | 1505 | decoder" should reject them just like malformed sequences for two | 1506 | reasons: (1) It helps to debug applications if overlong sequences are | 1507 | not treated as valid representations of characters, because this helps | 1508 | to spot problems more quickly. (2) Overlong sequences provide | 1509 | alternative representations of characters, that could maliciously be | 1510 | used to bypass filters that check only for ASCII characters. For | 1511 | instance, a 2-byte encoded line feed (LF) would not be caught by a | 1512 | line counter that counts only 0x0a bytes, but it would still be | 1513 | processed as a line feed by an unsafe UTF-8 decoder later in the | 1514 | pipeline. From a security point of view, ASCII compatibility of UTF-8 | 1515 | sequences means also, that ASCII characters are *only* allowed to be | 1516 | represented by ASCII bytes in the range 0x00-0x7f. To ensure this | 1517 | aspect of ASCII compatibility, use only "safe UTF-8 decoders" that | 1518 | reject overlong UTF-8 sequences for which a shorter encoding exists. | 1519 | | 1520 | 4.1 Examples of an overlong ASCII character | 1521 | | 1522 | With a safe UTF-8 decoder, all of the following five overlong | 1523 | representations of the ASCII character slash ("/") should be rejected | 1524 | like a malformed UTF-8 sequence, for instance by substituting it with | 1525 | a replacement character. If you see a slash below, you do not have a | 1526 | safe UTF-8 decoder! | 1527 | | 1528 | 4.1.1 U+002F = c0 af = "��" | 1529 | 4.1.2 U+002F = e0 80 af = "���" | 1530 | 4.1.3 U+002F = f0 80 80 af = "����" | 1531 | 4.1.4 U+002F = f8 80 80 80 af = "�����" | 1532 | 4.1.5 U+002F = fc 80 80 80 80 af = "������" | 1533 | | 1534 | 4.2 Maximum overlong sequences | 1535 | | 1536 | Below you see the highest Unicode value that is still resulting in an | 1537 | overlong sequence if represented with the given number of bytes. This | 1538 | is a boundary test for safe UTF-8 decoders. All five characters should | 1539 | be rejected like malformed UTF-8 sequences. | 1540 | | 1541 | 4.2.1 U-0000007F = c1 bf = "��" | 1542 | 4.2.2 U-000007FF = e0 9f bf = "���" | 1543 | 4.2.3 U-0000FFFF = f0 8f bf bf = "����" | 1544 | 4.2.4 U-001FFFFF = f8 87 bf bf bf = "�����" | 1545 | 4.2.5 U-03FFFFFF = fc 83 bf bf bf bf = "������" | 1546 | | 1547 | 4.3 Overlong representation of the NUL character | 1548 | | 1549 | The following five sequences should also be rejected like malformed | 1550 | UTF-8 sequences and should not be treated like the ASCII NUL | 1551 | character. | 1552 | | 1553 | 4.3.1 U+0000 = c0 80 = "��" | 1554 | 4.3.2 U+0000 = e0 80 80 = "���" | 1555 | 4.3.3 U+0000 = f0 80 80 80 = "����" | 1556 | 4.3.4 U+0000 = f8 80 80 80 80 = "�����" | 1557 | 4.3.5 U+0000 = fc 80 80 80 80 80 = "������" | 1558 | | 1559 | 5 Illegal code positions | 1560 | | 1561 | The following UTF-8 sequences should be rejected like malformed | 1562 | sequences, because they never represent valid ISO 10646 characters and | 1563 | a UTF-8 decoder that accepts them might introduce security problems | 1564 | comparable to overlong UTF-8 sequences. | 1565 | | 1566 | 5.1 Single UTF-16 surrogates | 1567 | | 1568 | 5.1.1 U+D800 = ed a0 80 = "���" | 1569 | 5.1.2 U+DB7F = ed ad bf = "���" | 1570 | 5.1.3 U+DB80 = ed ae 80 = "���" | 1571 | 5.1.4 U+DBFF = ed af bf = "���" | 1572 | 5.1.5 U+DC00 = ed b0 80 = "���" | 1573 | 5.1.6 U+DF80 = ed be 80 = "���" | 1574 | 5.1.7 U+DFFF = ed bf bf = "���" | 1575 | | 1576 | 5.2 Paired UTF-16 surrogates | 1577 | | 1578 | 5.2.1 U+D800 U+DC00 = ed a0 80 ed b0 80 = "������" | 1579 | 5.2.2 U+D800 U+DFFF = ed a0 80 ed bf bf = "������" | 1580 | 5.2.3 U+DB7F U+DC00 = ed ad bf ed b0 80 = "������" | 1581 | 5.2.4 U+DB7F U+DFFF = ed ad bf ed bf bf = "������" | 1582 | 5.2.5 U+DB80 U+DC00 = ed ae 80 ed b0 80 = "������" | 1583 | 5.2.6 U+DB80 U+DFFF = ed ae 80 ed bf bf = "������" | 1584 | 5.2.7 U+DBFF U+DC00 = ed af bf ed b0 80 = "������" | 1585 | 5.2.8 U+DBFF U+DFFF = ed af bf ed bf bf = "������" | 1586 | | 1587 | 5.3 Noncharacter code positions | 1588 | | 1589 | The following "noncharacters" are "reserved for internal use" by | 1590 | applications, and according to older versions of the Unicode Standard | 1591 | "should never be interchanged". Unicode Corrigendum #9 dropped the | 1592 | latter restriction. Nevertheless, their presence in incoming UTF-8 data | 1593 | can remain a potential security risk, depending on what use is made of | 1594 | these codes subsequently. Examples of such internal use: | 1595 | | 1596 | - Some file APIs with 16-bit characters may use the integer value -1 | 1597 | = U+FFFF to signal an end-of-file (EOF) or error condition. | 1598 | | 1599 | - In some UTF-16 receivers, code point U+FFFE might trigger a | 1600 | byte-swap operation (to convert between UTF-16LE and UTF-16BE). | 1601 | | 1602 | With such internal use of noncharacters, it may be desirable and safer | 1603 | to block those code points in UTF-8 decoders, as they should never | 1604 | occur legitimately in incoming UTF-8 data, and could trigger unsafe | 1605 | behaviour in subsequent processing. | 1606 | | 1607 | Particularly problematic noncharacters in 16-bit applications: | 1608 | | 1609 | 5.3.1 U+FFFE = ef bf be = "￾" | 1610 | 5.3.2 U+FFFF = ef bf bf = "￿" | 1611 | | 1612 | Other noncharacters: | 1613 | | 1614 | 5.3.3 U+FDD0 .. U+FDEF = "﷐﷑﷒﷓﷔﷕﷖﷗﷘﷙﷚﷛﷜﷝﷞﷟﷠﷡﷢﷣﷤﷥﷦﷧﷨﷩﷪﷫﷬﷭﷮﷯"| 1615 | | 1616 | 5.3.4 U+nFFFE U+nFFFF (for n = 1..10) | 1617 | | 1618 | "🿾🿿𯿾𯿿𿿾𿿿񏿾񏿿񟿾񟿿񯿾񯿿񿿾񿿿򏿾򏿿 | 1619 | 򟿾򟿿򯿾򯿿򿿾򿿿󏿾󏿿󟿾󟿿󯿾󯿿󿿾󿿿􏿾􏿿" | 1620 | | 1621 | THE END | 1622 | --------------------------------------------------------------------------------