![](\"https://github.com/Multiomics-Analytics-Group/course_protein_language_modeling/blob/main/img/nb_logo.png?raw=1\"){kind=logo}
"
9 | ]
10 | },
11 | {
12 | "cell_type": "markdown",
13 | "id": "8bf874eb-8949-41dd-a33b-0e55ba2954c6",
14 | "metadata": {},
15 | "source": [
16 | "[](https://colab.research.google.com/github/Multiomics-Analytics-Group/course_protein_language_modeling/blob/main/notebooks/embeddings.ipynb)\n",
17 | "\n",
18 | "\n",
19 | "This is a version of the notebook from [SETH](https://github.com/DagmarIlz/SETH) --- [here](https://colab.research.google.com/drive/1vDWh5YI_BPxQg0ku6CxKtSXEJ25u2wSq?usp=sharing)."
20 | ]
21 | },
22 | {
23 | "cell_type": "code",
24 | "execution_count": null,
25 | "id": "cd6f0c8f-d9db-4623-a2ee-483008dc9d74",
26 | "metadata": {},
27 | "outputs": [],
28 | "source": [
29 | "# uncomment the below to install dependencies for colab\n",
30 | "#!pip install \"transformers[torch]\" sentencepiece h5py biopython > /dev/null"
31 | ]
32 | },
33 | {
34 | "cell_type": "markdown",
35 | "id": "4efcc6ce-9843-4541-a6de-6b3a6ba14c45",
36 | "metadata": {},
37 | "source": [
38 | "Important
\n", 39 | "If you are running in Google Colab, change the Notebook settings to use `GPU`.\n", 40 | "\n", 41 | "Just follow **Edit** > **Notebook settings** or **Runtime** > **Change runtime type** and select **GPU** as Hardware accelerator.\n", 42 | "\n", 43 | "\n" 44 | ] 45 | }, 46 | { 47 | "cell_type": "markdown", 48 | "id": "75061fa6-eea7-42dc-b057-5da1c56094c0", 49 | "metadata": { 50 | "id": "5c0749e1" 51 | }, 52 | "source": [ 53 | "# Embedding Protein Sequences\n", 54 | "\n", 55 | "In this notebook, we will use a pre-trained language model, [ProtT5-XL-UniRef50](https://huggingface.co/Rostlab/prot_t5_xl_uniref50), to encode the protein sequences of 5000+ $\\beta$-$lactamase$ TEM-type varients from FASTA file [P62593.fasta](https://github.com/facebookresearch/esm/blob/2b369911bb5b4b0dda914521b9475cad1656b2ac/examples/data/P62593.fasta). This data was subsetted from a deep mutational scan released by [Gray et al. (2018)](https://www.cell.com/cell-systems/pdfExtended/S2405-4712(17)30492-1). \n", 56 | "\n", 57 | "The goal of this notebook is to obtain an embedding (fixed-dimensional vector representation) for each mutated sequence.\n", 58 | "\n", 59 | "Although the embedding won't capture all the information from the original data, good embedding representations allow us to analyze, cluster, or use them as features to train machine learning models. \n", 60 | "\n", 61 | "The embeddings generated in this notebook will then be used in the next exercise (prediction.ipynb) to train a simple varient predictor (i.e., predict the activity of the protein mutation)." 62 | ] 63 | }, 64 | { 65 | "cell_type": "markdown", 66 | "id": "38b6f6b9-0778-4435-a515-e9616522c4b3", 67 | "metadata": {}, 68 | "source": [ 69 | "\n",
70 | "\n",
71 | "
"
80 | ]
81 | },
82 | {
83 | "cell_type": "markdown",
84 | "id": "0e5c4fc2-00ea-4e24-90fa-d96bf3751810",
85 | "metadata": {
86 | "id": "5c0749e1"
87 | },
88 | "source": [
89 | "----\n",
90 | "\n",
91 | "## The Data: P62593 Sequences\n",
92 | "\n",
93 | "To start we will import and explore the dataset: "
94 | ]
95 | },
96 | {
97 | "cell_type": "code",
98 | "execution_count": null,
99 | "id": "e520014b-391d-41b0-ad62-7d51f56511b8",
100 | "metadata": {},
101 | "outputs": [],
102 | "source": [
103 | "# Set up working directories and download files/checkpoints \n",
104 | "!mkdir protT5 # directory for storing checkpoints, results etc\n",
105 | "!mkdir protT5/output # directory for storing your embeddings\n",
106 | "!curl -o P62593.fasta https://dl.fbaipublicfiles.com/fair-esm/examples/P62593.fasta"
107 | ]
108 | },
109 | {
110 | "cell_type": "code",
111 | "execution_count": null,
112 | "id": "bcfc1b48-24d6-4d6f-86aa-9d3048a6d1d0",
113 | "metadata": {},
114 | "outputs": [],
115 | "source": [
116 | "# Import dependencies\n",
117 | "from transformers import T5EncoderModel, T5Tokenizer\n",
118 | "import torch\n",
119 | "import h5py\n",
120 | "import time\n",
121 | "from Bio import SeqIO\n",
122 | "\n",
123 | "# Path variables\n",
124 | "per_protein_path = \"./protT5/output/per_protein_embeddings.h5\" # where to store the embeddings\n",
125 | "seq_path = 'P62593.fasta' # where the fasta file is saved\n",
126 | "\n",
127 | "# check whether GPU is available\n",
128 | "device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')\n",
129 | "print(f\"Using {device}\")"
130 | ]
131 | },
132 | {
133 | "cell_type": "code",
134 | "execution_count": null,
135 | "id": "009ac9f9-b169-4696-abf6-a1c0a3ef8edf",
136 | "metadata": {},
137 | "outputs": [],
138 | "source": [
139 | "def read_fasta(fasta_path:str) -> dict:\n",
140 | " '''\n",
141 | " reads in fasta file and returns a dictionary with primary id/sequence key/value pairs \n",
142 | " '''\n",
143 | " \n",
144 | " # dictionary to append to\n",
145 | " seqs = {}\n",
146 | " \n",
147 | " # read in and parse fasta file\n",
148 | " with open(fasta_path) as handle:\n",
149 | " for record in SeqIO.parse(handle, \"fasta\"):\n",
150 | " # append each varient to the dict\n",
151 | " seqs[record.id] = record.seq\n",
152 | "\n",
153 | " # verbose\n",
154 | " example_id=next(iter(seqs))\n",
155 | " print(f\"Read {len(seqs)} sequences.\")\n",
156 | " print(f\"Example:\\nKey: {example_id}\\nValue: {seqs[example_id]}\")\n",
157 | "\n",
158 | " return seqs"
159 | ]
160 | },
161 | {
162 | "cell_type": "code",
163 | "execution_count": null,
164 | "id": "3aa98336-a298-4f5f-87b6-98718ec5e22f",
165 | "metadata": {},
166 | "outputs": [],
167 | "source": [
168 | "# read in file\n",
169 | "fasta_output = read_fasta(seq_path)"
170 | ]
171 | },
172 | {
173 | "cell_type": "markdown",
174 | "id": "717503ab-a593-4af2-8128-4e64491c3ebc",
175 | "metadata": {},
176 | "source": [
177 | "In the FASTA file there are 5,397 sequences. As we can see in the example above from our fasta dictionary, each entry contains:\n",
178 | "\n",
179 | "- key: `{index}|beta-lactamase_{mutation}|{scaled_varient_effect}`\n",
180 | " > in prediction.ipynb we will be predicting the `scaled_varient_effect` value, which describes the scaled effect of the mutation. \n",
181 | "- value: the mutated $\\beta$-lactamase sequence, where a single residue is mutated (swapped with another amino acid)"
182 | ]
183 | },
184 | {
185 | "cell_type": "markdown",
186 | "id": "ef06a40d-a52f-46bf-bfe7-3b2374eea224",
187 | "metadata": {},
188 | "source": [
189 | "## The Model: ProtT5-XL-UniRef50\n",
190 | "\n",
191 | "ProtT5-XL-UniRef50 is based on the t5-3b model and was pretrained on a large corpus of protein sequences in a self-supervised fashion. This means it was pretrained on the raw protein sequences only, with **no humans-in-the-loop labelling** them in any way (which is why it can use lots of publicly available data) with an automatic process to generate inputs and labels from those protein sequences.\n",
192 | "\n",
193 | "This model only contains the encoder portion of the original ProtT5-XL-UniRef50 model using half precision (float16). As such, this model can efficiently be used to create protein/ amino acid representations. When used for training downstream networks/ feature extraction, these embeddings produced the same performance (established empirically by comparing on several downstream tasks). \n",
194 | "\n",
195 | "In the following cells we will prepare functions that will later assist us when generating the embeddings. "
196 | ]
197 | },
198 | {
199 | "cell_type": "markdown",
200 | "id": "d072fe5a-3d34-475b-b025-abb7f3eef1f8",
201 | "metadata": {},
202 | "source": [
203 | "### `get_T5_model()` Load encoder-part of ProtT5 in half-precision\n",
204 | "\n",
205 | "To start we create a function that will load the model and associated tokenizer. Recall from the previous notebook (model_training.ipynb) where every model on `transformers` comes with an associated `tokenizer` that handles tokenization for it, where tokenization for protein language models involve coverting each amino acid to a single token.\n",
206 | "\n",
207 | "**Recall: Fine-tuning flow chart from the previous notebook**\n",
208 | "\n",
209 | "\n",
210 | "\n",
211 | "This function accomplishes the \"Model Checkpoint Loading\" step. "
212 | ]
213 | },
214 | {
215 | "cell_type": "code",
216 | "execution_count": null,
217 | "id": "ac1c2e47-9131-41f6-b39b-2da11b4afe2f",
218 | "metadata": {},
219 | "outputs": [],
220 | "source": [
221 | "# Load ProtT5 in half-precision (more specifically: the encoder-part of ProtT5-XL-U50) \n",
222 | "def get_T5_model():\n",
223 | " '''\n",
224 | " retrieves the model and tokenizer\n",
225 | " '''\n",
226 | " # specify the encorder-part of the model \n",
227 | " model_checkpoint = 'Rostlab/prot_t5_xl_half_uniref50-enc'\n",
228 | " \n",
229 | " # import the model \n",
230 | " model = T5EncoderModel.from_pretrained(model_checkpoint)\n",
231 | " \n",
232 | " model = model.to(device) # move model to GPU\n",
233 | " model = model.eval() # set model to evaluation model\n",
234 | " \n",
235 | " # import tokenizer\n",
236 | " tokenizer = T5Tokenizer.from_pretrained(model_checkpoint)\n",
237 | "\n",
238 | " return model, tokenizer"
239 | ]
240 | },
241 | {
242 | "cell_type": "markdown",
243 | "id": "4404313f-0c46-4cae-acea-5bfdbe86bd99",
244 | "metadata": {},
245 | "source": [
246 | "Let's use the function:"
247 | ]
248 | },
249 | {
250 | "cell_type": "code",
251 | "execution_count": null,
252 | "id": "5282b4cc-36f4-4c70-93f2-5fb32a67f6b9",
253 | "metadata": {},
254 | "outputs": [],
255 | "source": [
256 | "# loading model\n",
257 | "model, tokenizer = get_T5_model()"
258 | ]
259 | },
260 | {
261 | "cell_type": "markdown",
262 | "id": "6ef88997-d8c8-4c62-b394-1d2c3b005a4f",
263 | "metadata": {},
264 | "source": [
265 | "### `get_embeddings()` Using the model to generate the embeddings\n",
266 | "\n",
267 | "From the flow chart above, the function `get_embeddings()` includes the 'Tokenization' and 'Dataset Creation' steps. Additionally, the model will encode the sequences. "
268 | ]
269 | },
270 | {
271 | "cell_type": "code",
272 | "execution_count": null,
273 | "id": "261698df-2497-4f6d-89ef-1df09736110a",
274 | "metadata": {},
275 | "outputs": [],
276 | "source": [
277 | "def get_embeddings(model, tokenizer, seqs, max_seq_len=1000, max_batch=100):\n",
278 | " '''\n",
279 | " use the encoder to embbed the sequences via batch-processing\n",
280 | " -----\n",
281 | " \n",
282 | " parameters:\n",
283 | " model: from get_T5_model()\n",
284 | " tokenizer: from get_T5_model()\n",
285 | " seqs: the dictionary of sequences generated by read_fasta() \n",
286 | " max_seq_length: the upper sequences length for applying batch-processing\n",
287 | " max_batch: the upper number of sequences per batch\n",
288 | " \n",
289 | " returns:\n",
290 | " results: a dictionary containing the embedding representations of the sequences\n",
291 | " '''\n",
292 | "\n",
293 | " # initialize a dictionary, the embeddings will be accessible from results['protein_embs'] \n",
294 | " results = {\"protein_embs\" : dict()}\n",
295 | "\n",
296 | " # sort sequences according to length (reduces unnecessary padding --> speeds up embedding)\n",
297 | " seq_dict = sorted(seqs.items(), \n",
298 | " # 'key' option is a function that serves as a basis of sort comparison.\n",
299 | " key=lambda kv: len(seqs[kv[0]]), \n",
300 | " # sort by descending order\n",
301 | " reverse=True\n",
302 | " )\n",
303 | " \n",
304 | " # for time tracking\n",
305 | " start = time.time()\n",
306 | " \n",
307 | " # initialize empty list\n",
308 | " batch = list()\n",
309 | " \n",
310 | " # for each item in the dictionary\n",
311 | " for seq_idx, (pdb_id, seq) in enumerate(seq_dict, 1):\n",
312 | " \n",
313 | " # add space between residues\n",
314 | " seq = ' '.join(list(seq))\n",
315 | " \n",
316 | " # length of sequence with spaces\n",
317 | " seq_len = len(seq)\n",
318 | " \n",
319 | " # append to batch list as tuple\n",
320 | " batch.append((pdb_id, seq, seq_len))\n",
321 | "\n",
322 | " # creates n-tuple pairs from each element in batch\n",
323 | " pdb_ids, seqs, seq_lens = zip(*batch)\n",
324 | " \n",
325 | " # empty list\n",
326 | " batch = list()\n",
327 | " \n",
328 | " # Data Preparation and Tokenization:\n",
329 | "\n",
330 | " # add_special_tokens adds extra token at the end of each sequence\n",
331 | " token_encoding = tokenizer.batch_encode_plus(seqs, add_special_tokens=True, padding=\"longest\")\n",
332 | " \n",
333 | " # making the tokenized sequence into a tensor\n",
334 | " input_ids = torch.tensor(token_encoding['input_ids']).to(device)\n",
335 | " \n",
336 | " # now making the mask into a tensor\n",
337 | " attention_mask = torch.tensor(token_encoding['attention_mask']).to(device)\n",
338 | " \n",
339 | " # Generate Embedding:\n",
340 | " \n",
341 | " # using the model to encode the sequence, generating an embedding representation\n",
342 | " try:\n",
343 | " with torch.no_grad():\n",
344 | " embedding_repr = model(input_ids, attention_mask=attention_mask)\n",
345 | " # verbosity for progress tracking\n",
346 | " print(f'Currently, embedding {pdb_id}')\n",
347 | " except RuntimeError:\n",
348 | " print(\"RuntimeError during embedding for {} (L={})\".format(pdb_id, seq_len))\n",
349 | " continue\n",
350 | " \n",
351 | " # Putting together the dataset:\n",
352 | " \n",
353 | " # slice off padding if any \n",
354 | " emb = embedding_repr.last_hidden_state[0,:seq_len]\n",
355 | "\n",
356 | " # average along column\n",
357 | " protein_emb = emb.mean(dim=0)\n",
358 | "\n",
359 | " # save the embedding into results dictionary where the key = the fasta file entry header\n",
360 | " results[\"protein_embs\"][pdb_id] = protein_emb.detach().cpu().numpy().squeeze()\n",
361 | "\n",
362 | " # get time elapsed\n",
363 | " passed_time=time.time()-start\n",
364 | " avg_time = passed_time/len(results[\"protein_embs\"])\n",
365 | " \n",
366 | " # final verbose\n",
367 | " print('\\n############# EMBEDDING STATS #############')\n",
368 | " print('Total number of per-protein embeddings: {}'.format(len(results[\"protein_embs\"])))\n",
369 | " print(\"Time for generating embeddings: {:.1f}[m] ({:.3f}[s/protein])\".format(\n",
370 | " passed_time/60, avg_time ))\n",
371 | " print('\\n############# END #############')\n",
372 | " \n",
373 | " return results"
374 | ]
375 | },
376 | {
377 | "cell_type": "markdown",
378 | "id": "53ccafd2-6032-4e50-be6a-53e98897f922",
379 | "metadata": {},
380 | "source": [
381 | "**NOTE: What are special tokens?** \n",
382 | "\n",
383 | "Special tokens aren't present in the input text, but carry important meaning that we want the model to act on. For exmaple (not spevific to our model): \n",
384 | "- [PAD] Padding token — Added to the end of shorter inputs so that all inputs have the same length. This is because inputs to a neural network model are typically batched and the model operates on entire batches. \n",
385 | "- [UNK] Unknown token — Used to limit the number of distinct tokens. For example, if we want a vocabulary of at most 1000 tokens but the input text has 1200, then the remaining 200 will be converted to [UNK].\n",
386 | " \n",
387 | "You can read more [here](https://medium.com/@alexkubiesa/special-tokens-in-tensorflow-3c7718dcb0ef).\n",
388 | "\n",
389 | "We will move on and use the function to get the embeddings:"
390 | ]
391 | },
392 | {
393 | "cell_type": "code",
394 | "execution_count": null,
395 | "id": "6f4097b1-8da4-4cd7-9be1-4caf817294ab",
396 | "metadata": {},
397 | "outputs": [],
398 | "source": [
399 | "# Compute embeddings\n",
400 | "results = get_embeddings(model, tokenizer, fasta_output)"
401 | ]
402 | },
403 | {
404 | "cell_type": "markdown",
405 | "id": "fc2e3350-2791-4773-99f2-e55113f288ea",
406 | "metadata": {},
407 | "source": [
408 | "### `save_embeddings()` Writing the embeddings to a file\n",
409 | "\n",
410 | "For our final function, we will write the embeddings to a file. We will load this file into prediction.ipynb to train machine learning models. This is also a copy of the embedding file in the reposition in _data/_. "
411 | ]
412 | },
413 | {
414 | "cell_type": "code",
415 | "execution_count": null,
416 | "id": "6986186f-93f3-4efe-9597-3c60ce5059a5",
417 | "metadata": {},
418 | "outputs": [],
419 | "source": [
420 | "def save_embeddings(emb_dict:dict , out_path:str):\n",
421 | " '''\n",
422 | " takes the resulting embeddings from get_embeddings() and saves to a compressed h5 file\n",
423 | " -----\n",
424 | " \n",
425 | " parameters:\n",
426 | " emb_dict (dict): dictionary that is in results['protein_embs'] \n",
427 | " out_path (str): path and filename where the embeddings will be saved\n",
428 | " '''\n",
429 | " \n",
430 | " with h5py.File(str(out_path), \"w\") as hf:\n",
431 | " for sequence_id, embedding in emb_dict.items():\n",
432 | " hf.create_dataset(sequence_id, data=embedding)\n",
433 | " \n",
434 | " return None"
435 | ]
436 | },
437 | {
438 | "cell_type": "code",
439 | "execution_count": null,
440 | "id": "aea3a1d4-0091-4134-9773-0a901f644a12",
441 | "metadata": {},
442 | "outputs": [],
443 | "source": [
444 | "# write embeddings to file\n",
445 | "save_embeddings(results[\"protein_embs\"], per_protein_path)"
446 | ]
447 | }
448 | ],
449 | "metadata": {
450 | "kernelspec": {
451 | "display_name": "Python 3 (ipykernel)",
452 | "language": "python",
453 | "name": "python3"
454 | },
455 | "language_info": {
456 | "codemirror_mode": {
457 | "name": "ipython",
458 | "version": 3
459 | },
460 | "file_extension": ".py",
461 | "mimetype": "text/x-python",
462 | "name": "python",
463 | "nbconvert_exporter": "python",
464 | "pygments_lexer": "ipython3",
465 | "version": "3.10.9"
466 | }
467 | },
468 | "nbformat": 4,
469 | "nbformat_minor": 5
470 | }
471 |
--------------------------------------------------------------------------------
/notebooks/model_training.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "id": "3494f2e1-0e62-4d3d-a161-6cff27727f78",
6 | "metadata": {},
7 | "source": [
8 | "NOTE
\n", 72 | "\n", 73 | " Even when using GPU, embedding the protein sequences takes some time (~25mins) so to begin go ahead and run all cells of this notebook so that the process is started in the background as we review the notebook.\n", 74 | "
\n", 75 | "\n", 76 | " A shortcut to running all cells is going to the \"Runtime\" menu and selecting \"Run all\".\n", 77 | "\n", 78 | "
\n", 79 | ""
9 | ]
10 | },
11 | {
12 | "cell_type": "markdown",
13 | "id": "d183c1c8-5e35-4eec-9078-caccab5c4708",
14 | "metadata": {},
15 | "source": [
16 | "[](https://colab.research.google.com/github/Multiomics-Analytics-Group/course_protein_language_modeling/blob/main/notebooks/model_training.ipynb)\n",
17 | "\n",
18 | "\n",
19 | "This is a version of the notebook from [Matthew Carrigan](https://huggingface.co/Rocketknight1) --- [here](https://huggingface.co/blog/deep-learning-with-proteins)."
20 | ]
21 | },
22 | {
23 | "cell_type": "code",
24 | "execution_count": 1,
25 | "id": "a78e617c-c8a7-4715-88d4-06e31e947c4a",
26 | "metadata": {},
27 | "outputs": [],
28 | "source": [
29 | "# uncomment the below to install dependencies for colab\n",
30 | "#!pip install \"transformers[torch]\" datasets evaluate biopython"
31 | ]
32 | },
33 | {
34 | "attachments": {},
35 | "cell_type": "markdown",
36 | "id": "5a6eb017-b2dc-4dbe-8771-69f1443d89e3",
37 | "metadata": {},
38 | "source": [
39 | "Important
\n", 40 | "If you are running in Google Colab, change the Notebook settings to use `GPU`.\n", 41 | "\n", 42 | "Just follow **Edit** > **Notebook settings** or **Runtime** > **Change runtime type** and select **GPU** as Hardware accelerator.\n", 43 | "\n", 44 | "![](\"https://github.com/Multiomics-Analytics-Group/course_protein_language_modeling/blob/main/img/gpu.png?raw=1\")
\n"
45 | ]
46 | },
47 | {
48 | "cell_type": "markdown",
49 | "id": "85f3e1b8-5517-4016-b7db-f1bb3bb33d31",
50 | "metadata": {
51 | "id": "5c0749e1"
52 | },
53 | "source": [
54 | "# Fine-Tuning Protein Language Models"
55 | ]
56 | },
57 | {
58 | "cell_type": "markdown",
59 | "id": "1b38b001-2450-4792-8647-31bae0bca58e",
60 | "metadata": {
61 | "id": "1d81db83"
62 | },
63 | "source": [
64 | "In this notebook, we are going to use transfer learning to fine-tune a large, pre-trained protein language model to predict **Subcellular location** based on protein sequence. \n",
65 | "\n",
66 | "The specific model we are going to use is [ESM](https://github.com/facebookresearch/esm), which is the state-of-the-art protein language model at the time of writing (November 2022) and described in detail in [Lin et al, 2022](https://www.biorxiv.org/content/10.1101/2022.07.20.500902v1).\n",
67 | "\n",
68 | "There are several ESM-2 checkpoints with differing model sizes. **A checkpoint** is an intermediate dump of a model’s entire internal state (weights, learning rate, etc.) so that the framework can resume the training from this point whenever desired. \n",
69 | "\n",
70 | "Larger models will generally have better accuracy, but they require more GPU memory and will take much longer to train. The available ESM-2 checkpoints are:\n",
71 | "\n",
72 | "| Checkpoint name | Num layers | Num parameters |\n",
73 | "|------------------------------|----|----------|\n",
74 | "| `esm2_t48_15B_UR50D` | 48 | 15B |\n",
75 | "| `esm2_t36_3B_UR50D` | 36 | 3B |\n",
76 | "| `esm2_t33_650M_UR50D` | 33 | 650M |\n",
77 | "| `esm2_t30_150M_UR50D` | 30 | 150M |\n",
78 | "| `esm2_t12_35M_UR50D` | 12 | 35M |\n",
79 | "| `esm2_t6_8M_UR50D` | 6 | 8M |\n",
80 | "\n",
81 | "*Note*: larger checkpoints may be very difficult to train without a large cloud GPU like an A100 or H100, and the largest 15B parameter checkpoint will probably be impossible to train on **any** single GPU! Also, note that memory usage for attention during training will scale as `O(batch_size * num_layers * seq_len^2)`, so larger models on long sequences will use quite a lot of memory! We will use the `esm2_t12_35M_UR50D` checkpoint for this notebook, which should train on any Colab instance or modern GPU."
82 | ]
83 | },
84 | {
85 | "cell_type": "code",
86 | "execution_count": 2,
87 | "id": "2c447d8c-00ee-4a3b-b924-ccc36e865828",
88 | "metadata": {
89 | "id": "32e605a2"
90 | },
91 | "outputs": [],
92 | "source": [
93 | "model_checkpoint = \"facebook/esm2_t12_35M_UR50D\""
94 | ]
95 | },
96 | {
97 | "attachments": {},
98 | "cell_type": "markdown",
99 | "id": "2cbf0310-6664-4938-80cf-0d6cbe741f2e",
100 | "metadata": {
101 | "id": "a8e6ac19"
102 | },
103 | "source": [
104 | "# Fine-tuning\n",
105 | "\n",
106 | "![](\"https://github.com/Multiomics-Analytics-Group/course_protein_language_modeling/blob/main/img/fine-tuning.png?raw=1\")
"
107 | ]
108 | },
109 | {
110 | "cell_type": "markdown",
111 | "id": "a7f033aa-d819-4e16-86de-feecf3d7b17c",
112 | "metadata": {},
113 | "source": [
114 | "## Subcellular Location"
115 | ]
116 | },
117 | {
118 | "cell_type": "markdown",
119 | "id": "33d88492-5d1e-4f4a-b09d-0f0642f17105",
120 | "metadata": {
121 | "id": "c5bc122f"
122 | },
123 | "source": [
124 | "## Data Collection"
125 | ]
126 | },
127 | {
128 | "cell_type": "markdown",
129 | "id": "57191840-da48-42ae-a5b9-c2eb87d0835f",
130 | "metadata": {
131 | "id": "4c91d394"
132 | },
133 | "source": [
134 | "We will gather data from [UniProt](https://www.uniprot.org/) to create a pair of lists: `sequences` and `labels`. `sequences` will be a list of protein sequences and `labels` will be a list of the category for each sequence (integers)."
135 | ]
136 | },
137 | {
138 | "cell_type": "code",
139 | "execution_count": 3,
140 | "id": "410bc787-ca60-44f5-98e6-1cb23e7e5088",
141 | "metadata": {
142 | "id": "c718ffbc"
143 | },
144 | "outputs": [
145 | {
146 | "name": "stdout",
147 | "output_type": "stream",
148 | "text": [
149 | "https://rest.uniprot.org/uniprotkb/stream?compressed=true&fields=accession%2Csequence%2Ccc_subcellular_location&format=tsv&query=%28%28organism_id%3A9606%29+AND+%28reviewed%3Atrue%29+AND+%28length%3A%5B80+TO+500%5D%29%29\n"
150 | ]
151 | }
152 | ],
153 | "source": [
154 | "import requests\n",
155 | "import urllib.parse\n",
156 | "\n",
157 | "organism = 9606\n",
158 | "fields = \",\".join(['accession', 'sequence', 'cc_subcellular_location'])\n",
159 | "query = f'((organism_id:{organism}) AND (reviewed:true) AND (length:[80 TO 500]))'\n",
160 | "params = {\n",
161 | " 'compressed': 'true', \n",
162 | " 'fields': fields, \n",
163 | " 'format':'tsv', \n",
164 | " 'query':query\n",
165 | "}\n",
166 | "query_url = 'https://rest.uniprot.org/uniprotkb/stream?'\n",
167 | "query_url += urllib.parse.urlencode(params)\n",
168 | "\n",
169 | "# take a look \n",
170 | "print(query_url)"
171 | ]
172 | },
173 | {
174 | "cell_type": "markdown",
175 | "id": "3412dd8d-958c-42da-bc26-0f021c832b7b",
176 | "metadata": {
177 | "id": "3d2edc14"
178 | },
179 | "source": [
180 | "We searched for `(organism_id:9606) AND (reviewed:true) AND (length:[80 TO 500])` on UniProt to get a list of reasonably-sized human proteins, then selected the format to TSV and the columns to `Accession`, `Sequence` and `Subcellular location [CC]`, since those contain the data we care about for this task."
181 | ]
182 | },
183 | {
184 | "cell_type": "code",
185 | "execution_count": 4,
186 | "id": "0bae78eb-e690-4574-b110-16f4f6a8a2e3",
187 | "metadata": {
188 | "id": "dd03ef98"
189 | },
190 | "outputs": [],
191 | "source": [
192 | "uniprot_request = requests.get(query_url)"
193 | ]
194 | },
195 | {
196 | "cell_type": "markdown",
197 | "id": "7494ca5e-f712-4b7b-9445-043942bc78a4",
198 | "metadata": {
199 | "id": "b7217b77"
200 | },
201 | "source": [
202 | "To get this data into Pandas, we use a `BytesIO` object, which Pandas will treat like a file. If you downloaded the data as a file you can skip this bit and just pass the filepath directly to `read_csv`."
203 | ]
204 | },
205 | {
206 | "cell_type": "code",
207 | "execution_count": 5,
208 | "id": "6e2481a9-b84b-420b-8fab-aadddc5160ef",
209 | "metadata": {
210 | "colab": {
211 | "base_uri": "https://localhost:8080/",
212 | "height": 468
213 | },
214 | "id": "f2c05017",
215 | "outputId": "9de817df-8a18-4ac3-b70b-452d8f6aa71a"
216 | },
217 | "outputs": [
218 | {
219 | "data": {
220 | "text/html": [
221 | "\n",
222 | "\n",
235 | "\n",
236 | " \n",
237 | "
\n",
277 | "
"
278 | ],
279 | "text/plain": [
280 | " Entry Sequence \\\n",
281 | "0 A0A0K2S4Q6 MTQRAGAAMLPSALLLLCVPGCLTVSGPSTVMGAVGESLSVQCRYE... \n",
282 | "1 A0AVI4 MDSPEVTFTLAYLVFAVCFVFTPNEFHAAGLTVQNLLSGWLGSEDA... \n",
283 | "2 A0JLT2 MENFTALFGAQADPPPPPTALGFGPGKPPPPPPPPAGGGPGTAPPP... \n",
284 | "3 A0M8Q6 GQPKAAPSVTLFPPSSEELQANKATLVCLVSDFNPGAVTVAWKADG... \n",
285 | "4 A0PJY2 MDSSCHNATTKMLATAPARGNMMSTSKPLAFSIERIMARTPEPKAL... \n",
286 | "\n",
287 | " Subcellular location [CC] \n",
288 | "0 SUBCELLULAR LOCATION: [Isoform 1]: Membrane {E... \n",
289 | "1 SUBCELLULAR LOCATION: Endoplasmic reticulum me... \n",
290 | "2 SUBCELLULAR LOCATION: Nucleus {ECO:0000305}. \n",
291 | "3 SUBCELLULAR LOCATION: Secreted {ECO:0000303|Pu... \n",
292 | "4 SUBCELLULAR LOCATION: Nucleus {ECO:0000269|Pub... "
293 | ]
294 | },
295 | "execution_count": 5,
296 | "metadata": {},
297 | "output_type": "execute_result"
298 | }
299 | ],
300 | "source": [
301 | "from io import BytesIO\n",
302 | "import pandas as pd\n",
303 | "\n",
304 | "# read in content to dataframe\n",
305 | "bio = BytesIO(uniprot_request.content)\n",
306 | "df = pd.read_csv(bio, compression='gzip', sep='\\t')\n",
307 | "\n",
308 | "# check out first 5 rows of df\n",
309 | "df.head()"
310 | ]
311 | },
312 | {
313 | "cell_type": "code",
314 | "execution_count": 6,
315 | "id": "ad5595c8-f8f1-42c3-aa53-cd332d7d38ac",
316 | "metadata": {},
317 | "outputs": [
318 | {
319 | "name": "stdout",
320 | "output_type": "stream",
321 | "text": [
322 | "| \n", 239 | " | Entry | \n", 240 | "Sequence | \n", 241 | "Subcellular location [CC] | \n", 242 | "
|---|---|---|---|
| 0 | \n", 247 | "A0A0K2S4Q6 | \n", 248 | "MTQRAGAAMLPSALLLLCVPGCLTVSGPSTVMGAVGESLSVQCRYE... | \n", 249 | "SUBCELLULAR LOCATION: [Isoform 1]: Membrane {E... | \n", 250 | "
| 1 | \n", 253 | "A0AVI4 | \n", 254 | "MDSPEVTFTLAYLVFAVCFVFTPNEFHAAGLTVQNLLSGWLGSEDA... | \n", 255 | "SUBCELLULAR LOCATION: Endoplasmic reticulum me... | \n", 256 | "
| 2 | \n", 259 | "A0JLT2 | \n", 260 | "MENFTALFGAQADPPPPPTALGFGPGKPPPPPPPPAGGGPGTAPPP... | \n", 261 | "SUBCELLULAR LOCATION: Nucleus {ECO:0000305}. | \n", 262 | "
| 3 | \n", 265 | "A0M8Q6 | \n", 266 | "GQPKAAPSVTLFPPSSEELQANKATLVCLVSDFNPGAVTVAWKADG... | \n", 267 | "SUBCELLULAR LOCATION: Secreted {ECO:0000303|Pu... | \n", 268 | "
| 4 | \n", 271 | "A0PJY2 | \n", 272 | "MDSSCHNATTKMLATAPARGNMMSTSKPLAFSIERIMARTPEPKAL... | \n", 273 | "SUBCELLULAR LOCATION: Nucleus {ECO:0000269|Pub... | \n", 274 | "
\n",
448 | "\n",
461 | "\n",
462 | " \n",
463 | "
\n",
503 | "
"
504 | ],
505 | "text/plain": [
506 | " Entry Sequence \\\n",
507 | "9 A1E959 MKIIILLGFLGATLSAPLIPQRLMSASNSNELLLNLNNGQLLPLQL... \n",
508 | "14 A1XBS5 MMRRTLENRNAQTKQLQTAVSNVEKHFGELCQIFAAYVRKTARLRD... \n",
509 | "18 A2RU49 MSSGNYQQSEALSKPTFSEEQASALVESVFGLKVSKVRPLPSYDDQ... \n",
510 | "20 A2RUH7 MEAATAPEVAAGSKLKVKEASPADAEPPQASPGQGAGSPTPQLLPP... \n",
511 | "21 A4D126 MEAGPPGSARPAEPGPCLSGQRGADHTASASLQSVAGTEPGRHPQA... \n",
512 | "\n",
513 | " Subcellular location [CC] \n",
514 | "9 SUBCELLULAR LOCATION: Secreted {ECO:0000250|Un... \n",
515 | "14 SUBCELLULAR LOCATION: Cytoplasm {ECO:0000269|P... \n",
516 | "18 SUBCELLULAR LOCATION: Cytoplasm {ECO:0000305}. \n",
517 | "20 SUBCELLULAR LOCATION: Cytoplasm, myofibril, sa... \n",
518 | "21 SUBCELLULAR LOCATION: Cytoplasm, cytosol {ECO:... "
519 | ]
520 | },
521 | "execution_count": 10,
522 | "metadata": {},
523 | "output_type": "execute_result"
524 | }
525 | ],
526 | "source": [
527 | "# use above booleans to filter df\n",
528 | "# NOTE: using .copy() when slicing/filtering an existing dataframe \n",
529 | "# prevents the triggering of 'SettingWithCopyWarning' \n",
530 | "cytosolic_df = df[cytosolic & ~membrane].copy()\n",
531 | "\n",
532 | "# check using .head() and/or .tail()\n",
533 | "cytosolic_df.head()"
534 | ]
535 | },
536 | {
537 | "cell_type": "code",
538 | "execution_count": 11,
539 | "id": "97b8ade4-d39e-48a1-ad5a-7908faeb9331",
540 | "metadata": {},
541 | "outputs": [
542 | {
543 | "data": {
544 | "text/html": [
545 | "| \n", 465 | " | Entry | \n", 466 | "Sequence | \n", 467 | "Subcellular location [CC] | \n", 468 | "
|---|---|---|---|
| 9 | \n", 473 | "A1E959 | \n", 474 | "MKIIILLGFLGATLSAPLIPQRLMSASNSNELLLNLNNGQLLPLQL... | \n", 475 | "SUBCELLULAR LOCATION: Secreted {ECO:0000250|Un... | \n", 476 | "
| 14 | \n", 479 | "A1XBS5 | \n", 480 | "MMRRTLENRNAQTKQLQTAVSNVEKHFGELCQIFAAYVRKTARLRD... | \n", 481 | "SUBCELLULAR LOCATION: Cytoplasm {ECO:0000269|P... | \n", 482 | "
| 18 | \n", 485 | "A2RU49 | \n", 486 | "MSSGNYQQSEALSKPTFSEEQASALVESVFGLKVSKVRPLPSYDDQ... | \n", 487 | "SUBCELLULAR LOCATION: Cytoplasm {ECO:0000305}. | \n", 488 | "
| 20 | \n", 491 | "A2RUH7 | \n", 492 | "MEAATAPEVAAGSKLKVKEASPADAEPPQASPGQGAGSPTPQLLPP... | \n", 493 | "SUBCELLULAR LOCATION: Cytoplasm, myofibril, sa... | \n", 494 | "
| 21 | \n", 497 | "A4D126 | \n", 498 | "MEAGPPGSARPAEPGPCLSGQRGADHTASASLQSVAGTEPGRHPQA... | \n", 499 | "SUBCELLULAR LOCATION: Cytoplasm, cytosol {ECO:... | \n", 500 | "
\n",
546 | "\n",
559 | "\n",
560 | " \n",
561 | "
\n",
601 | "
"
602 | ],
603 | "text/plain": [
604 | " Entry Sequence \\\n",
605 | "11527 Q8TDY3 MFNPHALDSPAVIFDNGSGFCKAGLSGEFGPRHMVSSIVGHLKFQA... \n",
606 | "11540 Q8WWF8 MAGTARHDREMAIQAKKKLTTATDPIERLRLQCLARGSAGIKGLGR... \n",
607 | "11665 Q9NUJ7 MGGQVSASNSFSRLHCRNANEDWMSALCPRLWDVPLHHLSIPGSHD... \n",
608 | "11667 Q9NVM6 MAVTKELLQMDLYALLGIEEKAADKEVKKAYRQKALSCHPDKNPDN... \n",
609 | "11675 Q9P2W6 MGRTWCGMWRRRRPGRRSAVPRWPHLSSQSGVEPPDRWTGTPGWPS... \n",
610 | "\n",
611 | " Subcellular location [CC] \n",
612 | "11527 SUBCELLULAR LOCATION: Cytoplasm, cytoskeleton ... \n",
613 | "11540 SUBCELLULAR LOCATION: Cytoplasm {ECO:0000305}. \n",
614 | "11665 SUBCELLULAR LOCATION: Cytoplasm {ECO:0000269|P... \n",
615 | "11667 SUBCELLULAR LOCATION: Cytoplasm {ECO:0000250|U... \n",
616 | "11675 SUBCELLULAR LOCATION: Cytoplasm. "
617 | ]
618 | },
619 | "execution_count": 11,
620 | "metadata": {},
621 | "output_type": "execute_result"
622 | }
623 | ],
624 | "source": [
625 | "cytosolic_df.tail()"
626 | ]
627 | },
628 | {
629 | "cell_type": "code",
630 | "execution_count": 12,
631 | "id": "ab5a996f-ca49-4f34-ac85-15b51ef09826",
632 | "metadata": {
633 | "colab": {
634 | "base_uri": "https://localhost:8080/",
635 | "height": 424
636 | },
637 | "id": "be5c420e",
638 | "outputId": "2588389b-05e0-4707-8e93-9995880b2dee"
639 | },
640 | "outputs": [
641 | {
642 | "data": {
643 | "text/html": [
644 | "| \n", 563 | " | Entry | \n", 564 | "Sequence | \n", 565 | "Subcellular location [CC] | \n", 566 | "
|---|---|---|---|
| 11527 | \n", 571 | "Q8TDY3 | \n", 572 | "MFNPHALDSPAVIFDNGSGFCKAGLSGEFGPRHMVSSIVGHLKFQA... | \n", 573 | "SUBCELLULAR LOCATION: Cytoplasm, cytoskeleton ... | \n", 574 | "
| 11540 | \n", 577 | "Q8WWF8 | \n", 578 | "MAGTARHDREMAIQAKKKLTTATDPIERLRLQCLARGSAGIKGLGR... | \n", 579 | "SUBCELLULAR LOCATION: Cytoplasm {ECO:0000305}. | \n", 580 | "
| 11665 | \n", 583 | "Q9NUJ7 | \n", 584 | "MGGQVSASNSFSRLHCRNANEDWMSALCPRLWDVPLHHLSIPGSHD... | \n", 585 | "SUBCELLULAR LOCATION: Cytoplasm {ECO:0000269|P... | \n", 586 | "
| 11667 | \n", 589 | "Q9NVM6 | \n", 590 | "MAVTKELLQMDLYALLGIEEKAADKEVKKAYRQKALSCHPDKNPDN... | \n", 591 | "SUBCELLULAR LOCATION: Cytoplasm {ECO:0000250|U... | \n", 592 | "
| 11675 | \n", 595 | "Q9P2W6 | \n", 596 | "MGRTWCGMWRRRRPGRRSAVPRWPHLSSQSGVEPPDRWTGTPGWPS... | \n", 597 | "SUBCELLULAR LOCATION: Cytoplasm. | \n", 598 | "
\n",
645 | "\n",
658 | "\n",
659 | " \n",
660 | "
\n",
700 | "
"
701 | ],
702 | "text/plain": [
703 | " Entry Sequence \\\n",
704 | "0 A0A0K2S4Q6 MTQRAGAAMLPSALLLLCVPGCLTVSGPSTVMGAVGESLSVQCRYE... \n",
705 | "3 A0M8Q6 GQPKAAPSVTLFPPSSEELQANKATLVCLVSDFNPGAVTVAWKADG... \n",
706 | "17 A2RU14 MAGTVLGVGAGVFILALLWVAVLLLCVLLSRASGAARFSVIFLFFG... \n",
707 | "33 A5X5Y0 MEGSWFHRKRFSFYLLLGFLLQGRGVTFTINCSGFGQHGADPTALN... \n",
708 | "36 A6ND01 MACWWPLLLELWTVMPTWAGDELLNICMNAKHHKRVPSPEDKLYEE... \n",
709 | "\n",
710 | " Subcellular location [CC] \n",
711 | "0 SUBCELLULAR LOCATION: [Isoform 1]: Membrane {E... \n",
712 | "3 SUBCELLULAR LOCATION: Secreted {ECO:0000303|Pu... \n",
713 | "17 SUBCELLULAR LOCATION: Membrane {ECO:0000305}; ... \n",
714 | "33 SUBCELLULAR LOCATION: Postsynaptic cell membra... \n",
715 | "36 SUBCELLULAR LOCATION: Cell membrane {ECO:00002... "
716 | ]
717 | },
718 | "execution_count": 12,
719 | "metadata": {},
720 | "output_type": "execute_result"
721 | }
722 | ],
723 | "source": [
724 | "# use boolean series to filter df\n",
725 | "membrane_df = df[membrane & ~cytosolic].copy()\n",
726 | "\n",
727 | "# check\n",
728 | "membrane_df.head()"
729 | ]
730 | },
731 | {
732 | "cell_type": "code",
733 | "execution_count": 13,
734 | "id": "7a0596b8-6655-4000-a8f6-555b961b84ab",
735 | "metadata": {},
736 | "outputs": [
737 | {
738 | "data": {
739 | "text/html": [
740 | "| \n", 662 | " | Entry | \n", 663 | "Sequence | \n", 664 | "Subcellular location [CC] | \n", 665 | "
|---|---|---|---|
| 0 | \n", 670 | "A0A0K2S4Q6 | \n", 671 | "MTQRAGAAMLPSALLLLCVPGCLTVSGPSTVMGAVGESLSVQCRYE... | \n", 672 | "SUBCELLULAR LOCATION: [Isoform 1]: Membrane {E... | \n", 673 | "
| 3 | \n", 676 | "A0M8Q6 | \n", 677 | "GQPKAAPSVTLFPPSSEELQANKATLVCLVSDFNPGAVTVAWKADG... | \n", 678 | "SUBCELLULAR LOCATION: Secreted {ECO:0000303|Pu... | \n", 679 | "
| 17 | \n", 682 | "A2RU14 | \n", 683 | "MAGTVLGVGAGVFILALLWVAVLLLCVLLSRASGAARFSVIFLFFG... | \n", 684 | "SUBCELLULAR LOCATION: Membrane {ECO:0000305}; ... | \n", 685 | "
| 33 | \n", 688 | "A5X5Y0 | \n", 689 | "MEGSWFHRKRFSFYLLLGFLLQGRGVTFTINCSGFGQHGADPTALN... | \n", 690 | "SUBCELLULAR LOCATION: Postsynaptic cell membra... | \n", 691 | "
| 36 | \n", 694 | "A6ND01 | \n", 695 | "MACWWPLLLELWTVMPTWAGDELLNICMNAKHHKRVPSPEDKLYEE... | \n", 696 | "SUBCELLULAR LOCATION: Cell membrane {ECO:00002... | \n", 697 | "
\n",
741 | "\n",
754 | "\n",
755 | " \n",
756 | "
\n",
796 | "
"
797 | ],
798 | "text/plain": [
799 | " Entry Sequence \\\n",
800 | "11896 Q86UQ5 MQSDIYHPGHSFPSWVLCWVHSCGHEGHLRETAEIRKTHQNGDLQI... \n",
801 | "11920 Q8N8V8 MLLKVRRASLKPPATPHQGAFRAGNVIGQLIYLLTWSLFTAWLRPP... \n",
802 | "11959 Q96N68 MQGQGALKESHIHLPTEQPEASLVLQGQLAESSALGPKGALRPQAQ... \n",
803 | "11966 Q9H0A3 MMNNTDFLMLNNPWNKLCLVSMDFCFPLDFVSNLFWIFASKFIIVT... \n",
804 | "11969 Q9H354 MNKHNLRLVQLASELILIEIIPKLFLSQVTTISHIKREKIPPNHRK... \n",
805 | "\n",
806 | " Subcellular location [CC] \n",
807 | "11896 SUBCELLULAR LOCATION: Membrane {ECO:0000305}; ... \n",
808 | "11920 SUBCELLULAR LOCATION: Membrane {ECO:0000305}; ... \n",
809 | "11959 SUBCELLULAR LOCATION: Membrane {ECO:0000305}; ... \n",
810 | "11966 SUBCELLULAR LOCATION: Membrane {ECO:0000255}; ... \n",
811 | "11969 SUBCELLULAR LOCATION: Membrane {ECO:0000305}; ... "
812 | ]
813 | },
814 | "execution_count": 13,
815 | "metadata": {},
816 | "output_type": "execute_result"
817 | }
818 | ],
819 | "source": [
820 | "membrane_df.tail()"
821 | ]
822 | },
823 | {
824 | "cell_type": "markdown",
825 | "id": "9798179a-96ac-42d3-959e-79cd9564a63c",
826 | "metadata": {
827 | "id": "77e8cea6"
828 | },
829 | "source": [
830 | "Now, let's add labels. We use `0` as the label for cytosolic proteins and `1` as the label for membrane proteins."
831 | ]
832 | },
833 | {
834 | "cell_type": "code",
835 | "execution_count": 14,
836 | "id": "4ff5bbdb-6f11-4d9e-995a-2019f1338b73",
837 | "metadata": {},
838 | "outputs": [],
839 | "source": [
840 | "# adding label columns\n",
841 | "cytosolic_df['label'] = 0\n",
842 | "membrane_df['label'] = 1"
843 | ]
844 | },
845 | {
846 | "cell_type": "markdown",
847 | "id": "e430cec6-fb0c-45db-8854-52e9d37014a2",
848 | "metadata": {
849 | "id": "5a4bbda2"
850 | },
851 | "source": [
852 | "Now we will combine the 2 groups to form our initial dataset. Don't worry - the proteins will get shuffled when splitting the data!"
853 | ]
854 | },
855 | {
856 | "cell_type": "code",
857 | "execution_count": 15,
858 | "id": "a589d77f-7c63-4b5c-b584-8a4ca8f3d4cb",
859 | "metadata": {},
860 | "outputs": [
861 | {
862 | "name": "stdout",
863 | "output_type": "stream",
864 | "text": [
865 | "| \n", 758 | " | Entry | \n", 759 | "Sequence | \n", 760 | "Subcellular location [CC] | \n", 761 | "
|---|---|---|---|
| 11896 | \n", 766 | "Q86UQ5 | \n", 767 | "MQSDIYHPGHSFPSWVLCWVHSCGHEGHLRETAEIRKTHQNGDLQI... | \n", 768 | "SUBCELLULAR LOCATION: Membrane {ECO:0000305}; ... | \n", 769 | "
| 11920 | \n", 772 | "Q8N8V8 | \n", 773 | "MLLKVRRASLKPPATPHQGAFRAGNVIGQLIYLLTWSLFTAWLRPP... | \n", 774 | "SUBCELLULAR LOCATION: Membrane {ECO:0000305}; ... | \n", 775 | "
| 11959 | \n", 778 | "Q96N68 | \n", 779 | "MQGQGALKESHIHLPTEQPEASLVLQGQLAESSALGPKGALRPQAQ... | \n", 780 | "SUBCELLULAR LOCATION: Membrane {ECO:0000305}; ... | \n", 781 | "
| 11966 | \n", 784 | "Q9H0A3 | \n", 785 | "MMNNTDFLMLNNPWNKLCLVSMDFCFPLDFVSNLFWIFASKFIIVT... | \n", 786 | "SUBCELLULAR LOCATION: Membrane {ECO:0000255}; ... | \n", 787 | "
| 11969 | \n", 790 | "Q9H354 | \n", 791 | "MNKHNLRLVQLASELILIEIIPKLFLSQVTTISHIKREKIPPNHRK... | \n", 792 | "SUBCELLULAR LOCATION: Membrane {ECO:0000305}; ... | \n", 793 | "
mean=%{marker.color}
mean=%{marker.color}