├── .gitignore ├── README.md ├── README.md~ ├── docs ├── activations.md ├── autograd.md ├── getting_started.md ├── layers.md ├── losses.md ├── optimizers.md ├── tensor.md └── transformers.md ├── examples ├── example1.lisp └── example2.lisp ├── generate_project_files.sh ├── neuralisp.asd ├── src ├── activations │ ├── base.lisp │ ├── relu.lisp │ └── sigmoid.lisp ├── core │ ├── autograd.lisp │ ├── gpu.lisp │ └── tensor.lisp ├── layers │ ├── attention.lisp │ ├── base.lisp │ ├── convolutional.lisp │ ├── linear.lisp │ ├── multihead_attention.lisp │ └── recurrent.lisp ├── losses │ ├── base.lisp │ ├── cross_entropy.lisp │ └── mse.lisp ├── optimizers │ ├── adam.lisp │ ├── base.lisp │ └── sgd.lisp ├── transformers │ ├── decoder.lisp │ ├── encoder.lisp │ └── transformer_layer.lisp └── utils │ ├── data_loader.lisp │ └── debugging.lisp └── tests ├── activations └── test_activations.lisp ├── core ├── test_autograd.lisp ├── test_gpu.lisp └── test_tensor.lisp ├── layers └── test_layers.lisp ├── losses └── test_losses.lisp ├── optimizers └── test_optimizers.lisp └── transformers └── test_transformers.lisp /.gitignore: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ck46/neuralisp/b9b594c0e83ea76cbb11598b7f495fbb97ba90ca/.gitignore -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # NeuraLisp 2 | 3 | NeuraLisp is a machine learning framework for Common Lisp, designed for ease of use and extensibility. It provides a modular implementation for building, training, and evaluating neural networks for various machine learning tasks. 4 | 5 | ## Features 6 | 7 | - Core components for tensor operations, autograd, GPU support, and more 8 | - A variety of layers, including linear, convolutional, recurrent, and attention-based layers 9 | - Common activation functions like ReLU, Sigmoid, and Tanh 10 | - Loss functions like mean squared error and cross-entropy 11 | - Optimizers like Stochastic Gradient Descent (SGD) and Adam 12 | 13 | ## Installation 14 | 15 | To install NeuraLisp, clone the repository and load the system definition using Quicklisp or ASDF: 16 | 17 | ``` 18 | git clone https://github.com/yourusername/NeuraLisp.git 19 | ``` 20 | 21 | Then, in your Common Lisp REPL: 22 | 23 | ```lisp 24 | (ql:quickload :NeuraLisp) 25 | ``` 26 | 27 | ## Usage 28 | 29 | Here's an example of how to create a simple neural network using NeuraLisp: 30 | 31 | ```lisp 32 | (use-package :NeuraLisp.core.tensor) 33 | (use-package :NeuraLisp.layers.linear) 34 | (use-package :NeuraLisp.activations.relu) 35 | 36 | ;; Create a linear layer 37 | (defvar *layer* (make-instance 'linear :input-dim 3 :output-dim 2)) 38 | 39 | ;; Create input tensor 40 | (defvar *input* (make-tensor #(1.0 2.0 3.0) #(1 3))) 41 | 42 | ;; Apply the linear layer 43 | (defvar *output* (forward *layer* *input*)) 44 | 45 | ;; Apply ReLU activation 46 | (defvar *relu* (make-instance 'relu)) 47 | (defvar *activated-output* (activate *relu* *output*)) 48 | ``` 49 | 50 | For more examples, please refer to the `examples/` folder. 51 | 52 | ## Documentation 53 | 54 | Detailed documentation for each component can be found in the `docs/` folder. 55 | 56 | ## Contributing 57 | 58 | We welcome contributions to NeuraLisp! If you find a bug, want to improve the code quality, or have ideas for new features, please feel free to create an issue or submit a pull request on GitHub. 59 | 60 | ## License 61 | 62 | NeuraLisp is licensed under the MIT License. Please see the `LICENSE` file for more information. 63 | -------------------------------------------------------------------------------- /README.md~: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ck46/neuralisp/b9b594c0e83ea76cbb11598b7f495fbb97ba90ca/README.md~ -------------------------------------------------------------------------------- /docs/activations.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ck46/neuralisp/b9b594c0e83ea76cbb11598b7f495fbb97ba90ca/docs/activations.md -------------------------------------------------------------------------------- /docs/autograd.md: -------------------------------------------------------------------------------- 1 | # Autograd 2 | 3 | The autograd module provides automatic differentiation and gradient computation for the Neuralisp machine learning framework. This module defines variable classes and functions for creating variables that hold tensor values, gradient tensors, and backward functions to compute changes in variables when updating the network. 4 | 5 | ## Usage 6 | 7 | To use the autograd module, import the functions and classes provided by the `neuralisp.core.autograd` package: 8 | 9 | ```common-lisp 10 | (use-package :neuralisp.core.autograd) 11 | ``` 12 | 13 | ## Classes 14 | 15 | ### `variable` 16 | 17 | The `variable` class represents an autograd variable and contains the following slots: 18 | 19 | - `value`: A tensor that represents the value of the variable. 20 | - `gradient`: A tensor (or null) that represents the gradient of the variable. 21 | - `backward`: A function (or null) used for computing the gradients during the backward pass. 22 | 23 | ## Functions 24 | 25 | ### `create-variable` (value &key (requires-grad t) (on-gpu nil)) 26 | 27 | This function creates a new autograd variable with the input value, a gradient tensor (if requires-grad is true), and sets the backward function to `nil`. If `on-gpu` is true, the created variable and its gradients will be moved to the GPU. 28 | 29 | Arguments: 30 | 31 | - `value`: A tensor representing the value of the new variable. 32 | - `requires-grad` (optional, default: `t`): If true, the variable's gradient tensor will be created. 33 | - `on-gpu` (optional, default: `nil`): If true, the variable's tensor and gradients will be moved to the GPU. 34 | 35 | Returns: 36 | 37 | - A new autograd `variable` instance. 38 | 39 | ### `backward` (var &optional (grad-output 1.0)) 40 | 41 | Computes the gradients of the variable with respect to its values and accumulates gradient output. 42 | 43 | Arguments: 44 | 45 | - `var`: An autograd `variable` to compute gradients for. 46 | - `grad-output` (optional, default: `1.0`): A scalar value for the gradient output's accumulation. 47 | 48 | ### `zero-gradient` (var) 49 | 50 | Sets the gradient of the variable to zero. 51 | 52 | Arguments: 53 | 54 | - `var`: An autograd `variable` instance whose gradient will be set to zero. 55 | 56 | ### `partial-grad` (node-a node-b) 57 | 58 | Computes the partial derivatives between two 'variable' nodes. 59 | 60 | Arguments: 61 | 62 | - `node-a` and `node-b`: `variable` nodes in the computational graph. 63 | 64 | Returns: 65 | 66 | - A scalar representing the computed partial gradients. 67 | 68 | ### `apply-partial-grad` (node-a node-b) 69 | 70 | Applies the computed partial gradients of node-b with respect to node-a to the gradients of both nodes. 71 | 72 | Arguments: 73 | 74 | - `node-a` and `node-b`: `variable` nodes in the computational graph. 75 | 76 | ## Examples 77 | 78 | Creating an autograd variable: 79 | 80 | ```common-lisp 81 | (defparameter *var* 82 | (create-variable (make-tensor (list 3 3) '(0.5d0 0.5d0)))) 83 | ``` 84 | 85 | Performing the backward pass on a variable: 86 | 87 | ```common-lisp 88 | ; Assuming *var* has a backward function assigned 89 | (backward *var*) 90 | ``` 91 | 92 | Setting the gradient of a variable to zero: 93 | 94 | ```common-lisp 95 | (zero-gradient *var*) 96 | ``` 97 | 98 | Computing and applying partial gradients between variables: 99 | 100 | ```common-lisp 101 | ; Assuming *var-a* and *var-b* are variables in the computational graph 102 | (defparameter *partial* 103 | (partial-grad *var-a* *var-b*)) 104 | 105 | (apply-partial-grad *var-a* *var-b*) 106 | ``` -------------------------------------------------------------------------------- /docs/getting_started.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ck46/neuralisp/b9b594c0e83ea76cbb11598b7f495fbb97ba90ca/docs/getting_started.md -------------------------------------------------------------------------------- /docs/layers.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ck46/neuralisp/b9b594c0e83ea76cbb11598b7f495fbb97ba90ca/docs/layers.md -------------------------------------------------------------------------------- /docs/losses.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ck46/neuralisp/b9b594c0e83ea76cbb11598b7f495fbb97ba90ca/docs/losses.md -------------------------------------------------------------------------------- /docs/optimizers.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ck46/neuralisp/b9b594c0e83ea76cbb11598b7f495fbb97ba90ca/docs/optimizers.md -------------------------------------------------------------------------------- /docs/tensor.md: -------------------------------------------------------------------------------- 1 | # Tensor 2 | 3 | The tensor module provides basic tensor operations and data structures for the Neuralisp machine learning framework. This module defines tensor classes and functions to create, manipulate and perform mathematical operations on multi-dimensional tensors, including support for GPU acceleration. 4 | 5 | ## Usage 6 | 7 | To use the tensor module, import the functions and classes provided by the `neuralisp.core.tensor` package: 8 | 9 | ```common-lisp 10 | (use-package :neuralisp.core.tensor) 11 | ``` 12 | 13 | ## Classes 14 | 15 | ### `tensor` 16 | 17 | The `tensor` class represents a multi-dimensional array (tensor) and contains the following slots: 18 | 19 | - `data`: A simple-array holding the numerical values of the tensor. 20 | - `shape`: A list of integers representing the dimensions of the tensor. 21 | 22 | ## Functions 23 | 24 | ### `make-tensor` (shape &key (initial-element 0) (on-gpu nil)) 25 | 26 | This function creates a new tensor with the specified shape and initializes to the given initial-element. If `on-gpu` is true, the created tensor will be moved to the GPU. 27 | 28 | Arguments: 29 | 30 | - `shape`: A list of integers representing the dimensions of the new tensor. 31 | - `initial-element` (optional, default: `0`): The initial value used to fill the tensor. 32 | - `on-gpu` (optional, default: `nil`): If true, the tensor will be moved to the GPU. 33 | 34 | Returns: 35 | 36 | - A new `tensor` instance. 37 | 38 | ## Tensor Operations 39 | 40 | The following functions perform element-wise operations on tensors. 41 | 42 | - `tensor-add` (tensor-a tensor-b) 43 | - `tensor-subtract` (tensor-a tensor-b) 44 | - `tensor-multiply` (tensor-a tensor-b) 45 | - `tensor-divide` (tensor-a tensor-b) 46 | 47 | ### Broadcasting 48 | 49 | When performing element-wise operations on tensors with different shapes, the tensor module automatically broadcasts the smaller tensor to match the shape of the larger tensor, if the shapes are compatible. 50 | 51 | ### Matrix Multiplication 52 | 53 | - `tensor-matmul` (tensor-a tensor-b &key (transpose-a nil) (transpose-b nil)) 54 | 55 | This function performs matrix multiplication between two tensors. 56 | 57 | Arguments: 58 | 59 | - `tensor-a` and `tensor-b`: Tensors to be multiplied 60 | - `transpose-a` (optional, default: `nil`): If true, tensor-a will be transposed before multiplying 61 | - `transpose-b` (optional, default: `nil`): If true, tensor-b will be transposed before multiplying 62 | 63 | Returns: 64 | 65 | - A new `tensor` instance representing the result of the multiplication. 66 | 67 | ### Reduction Operations 68 | 69 | - `tensor-sum` (tensor &key (axis nil) (keepdims nil)) 70 | - `tensor-mean` (tensor &key (axis nil) (keepdims nil)) 71 | 72 | These functions perform reduction operations on a tensor along the specified axis or axes. If no axis is specified, the reduction is applied across all elements of the tensor. 73 | 74 | Arguments: 75 | 76 | - `tensor`: A tensor to perform the reduction operation on 77 | - `axis` (optional, default: `nil`): An integer or list of integers representing the axis or axes to reduce 78 | - `keepdims` (optional, default: `nil`): If true, the reduced axes will be kept with size 1 79 | 80 | ## Examples 81 | 82 | Creating a tensor: 83 | 84 | ```common-lisp 85 | (defparameter *a* 86 | (make-tensor (list 2 3) :initial-element 0.5d0)) 87 | ``` 88 | 89 | Performing element-wise operations on tensors: 90 | 91 | ```common-lisp 92 | (defparameter *b* 93 | (tensor-add *a* *a*)) 94 | ``` 95 | 96 | Matrix multiplication: 97 | 98 | ```common-lisp 99 | (defparameter *c* 100 | (tensor-matmul *a* (tensor-transpose *a*))) 101 | ``` 102 | 103 | Performing reduction operations: 104 | 105 | ```common-lisp 106 | (defparameter *sum* 107 | (tensor-sum *a* :axis 0)) 108 | 109 | (defparameter *mean* 110 | (tensor-mean *a* :axis 1 :keepdims t)) 111 | ``` -------------------------------------------------------------------------------- /docs/transformers.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ck46/neuralisp/b9b594c0e83ea76cbb11598b7f495fbb97ba90ca/docs/transformers.md -------------------------------------------------------------------------------- /examples/example1.lisp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ck46/neuralisp/b9b594c0e83ea76cbb11598b7f495fbb97ba90ca/examples/example1.lisp -------------------------------------------------------------------------------- /examples/example2.lisp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ck46/neuralisp/b9b594c0e83ea76cbb11598b7f495fbb97ba90ca/examples/example2.lisp -------------------------------------------------------------------------------- /generate_project_files.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Create directories 4 | mkdir -p src/core 5 | mkdir -p src/layers 6 | mkdir -p src/activations 7 | mkdir -p src/optimizers 8 | mkdir -p src/losses 9 | mkdir -p src/transformers 10 | mkdir -p src/utils 11 | mkdir -p tests/core 12 | mkdir -p tests/layers 13 | mkdir -p tests/activations 14 | mkdir -p tests/optimizers 15 | mkdir -p tests/losses 16 | mkdir -p tests/transformers 17 | mkdir -p examples 18 | mkdir -p docs 19 | 20 | # Create files 21 | touch src/core/tensor.lisp 22 | touch src/core/gpu.lisp 23 | touch src/core/autograd.lisp 24 | touch src/layers/base.lisp 25 | touch src/layers/linear.lisp 26 | touch src/layers/convolutional.lisp 27 | touch src/layers/recurrent.lisp 28 | touch src/layers/attention.lisp 29 | touch src/layers/multihead_attention.lisp 30 | touch src/activations/base.lisp 31 | touch src/activations/relu.lisp 32 | touch src/activations/sigmoid.lisp 33 | touch src/optimizers/base.lisp 34 | touch src/optimizers/sgd.lisp 35 | touch src/optimizers/adam.lisp 36 | touch src/losses/base.lisp 37 | touch src/losses/mse.lisp 38 | touch src/losses/cross_entropy.lisp 39 | touch src/transformers/transformer_layer.lisp 40 | touch src/transformers/encoder.lisp 41 | touch src/transformers/decoder.lisp 42 | touch src/utils/data_loader.lisp 43 | touch src/utils/debugging.lisp 44 | touch tests/core/test_tensor.lisp 45 | touch tests/core/test_gpu.lisp 46 | touch tests/core/test_autograd.lisp 47 | touch tests/layers/test_layers.lisp 48 | touch tests/activations/test_activations.lisp 49 | touch tests/optimizers/test_optimizers.lisp 50 | touch tests/losses/test_losses.lisp 51 | touch tests/transformers/test_transformers.lisp 52 | touch examples/example1.lisp 53 | touch examples/example2.lisp 54 | touch docs/getting_started.md 55 | touch docs/tensor.md 56 | touch docs/autograd.md 57 | touch docs/layers.md 58 | touch docs/activations.md 59 | touch docs/optimizers.md 60 | touch docs/losses.md 61 | touch docs/transformers.md 62 | # touch README.md 63 | touch .gitignore 64 | touch neuralisp.asd 65 | 66 | # Git setup 67 | git init 68 | 69 | # Optional: Configure user details 70 | # (Replace "Your Name" and "your.email@example.com" with your information) 71 | git config user.name "ck46" 72 | git config user.email "prof.chakas@gmail.com" 73 | 74 | git add . 75 | git commit -m "Initial commit" -------------------------------------------------------------------------------- /neuralisp.asd: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ck46/neuralisp/b9b594c0e83ea76cbb11598b7f495fbb97ba90ca/neuralisp.asd -------------------------------------------------------------------------------- /src/activations/base.lisp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ck46/neuralisp/b9b594c0e83ea76cbb11598b7f495fbb97ba90ca/src/activations/base.lisp -------------------------------------------------------------------------------- /src/activations/relu.lisp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ck46/neuralisp/b9b594c0e83ea76cbb11598b7f495fbb97ba90ca/src/activations/relu.lisp -------------------------------------------------------------------------------- /src/activations/sigmoid.lisp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ck46/neuralisp/b9b594c0e83ea76cbb11598b7f495fbb97ba90ca/src/activations/sigmoid.lisp -------------------------------------------------------------------------------- /src/core/autograd.lisp: -------------------------------------------------------------------------------- 1 | (defpackage :neuralisp.core.autograd 2 | (:use :common-lisp) 3 | (:import-from :neuralisp.core.tensor :tensor :tensor-data :tensor-shape :make-tensor) 4 | (:import-from :neuralisp.core.gpu :move-to-gpu :move-to-cpu) 5 | (:export :variable :value :gradient :create-variable :backward :zero-gradient 6 | :partial-grad :apply-partial-grad)) 7 | (in-package :neuralisp.core.autograd) 8 | 9 | (defclass variable () 10 | ((value :initarg :value 11 | :reader variable-value 12 | :type tensor 13 | :documentation "The tensor object representing the value of the variable.") 14 | (gradient :initarg :gradient 15 | :accessor variable-gradient 16 | :type (or null tensor) 17 | :documentation "The tensor object representing the gradient of the variable.") 18 | (backward :initarg :backward 19 | :accessor variable-backward 20 | :type (or null function) 21 | :documentation "The backward function for computing gradients."))) 22 | 23 | (defun create-variable (value &key (requires-grad t) (on-gpu nil)) 24 | "Create a new autograd variable with the input value and a gradient, 25 | and set backward to nil." 26 | (let ((tensor-value (if on-gpu (move-to-gpu value) value)) 27 | (tensor-grad (when requires-grad 28 | (let ((grad-tensor (make-tensor (make-array (reduce #'* (tensor-shape value))) (tensor-shape value)))) 29 | (if on-gpu (move-to-gpu grad-tensor) grad-tensor))))) 30 | (make-instance 'variable :value tensor-value 31 | :gradient tensor-grad))) 32 | 33 | (defun backward (var &optional (grad-output 1.0)) 34 | "Computes the gradients of the variable with respect to its values, 35 | and accumulated gradient output." 36 | (when (functionp (variable-backward var)) 37 | (funcall (variable-backward var) grad-output))) 38 | 39 | (defun zero-gradient (var) 40 | "Set the gradient of the variable to zero." 41 | (setf (variable-gradient var) 42 | (let ((zero-tensor (make-instance 'tensor :data (make-array (reduce #'* (tensor-shape (variable-value var)))) :shape (tensor-shape (variable-value var))))) 43 | (if (equal (tensor-data (variable-value var)) :gpu) (move-to-gpu zero-tensor) zero-tensor)))) 44 | 45 | (defun partial-grad (node-a node-b) 46 | "Compute the partial derivatives between two 'variable' nodes." 47 | (let ((prev-node node-b) (grad 1)) 48 | (loop 49 | (when (eq prev-node node-a) 50 | (return grad)) 51 | (setq grad (* grad (variable-gradient prev-node))) 52 | (setq prev-node (variable-backward prev-node))))) 53 | 54 | (defun apply-partial-grad (node-a node-b) 55 | "Apply the computed partial gradients of node-b with respect to node-a to the gradients of both nodes." 56 | (let ((partial (partial-grad node-a node-b))) 57 | (setf (variable-gradient node-a) (* (variable-gradient node-a) partial)) 58 | (setf (variable-gradient node-b) (* (variable-gradient node-b) partial)))) -------------------------------------------------------------------------------- /src/core/gpu.lisp: -------------------------------------------------------------------------------- 1 | (defpackage :neuralisp.core.gpu 2 | (:use :common-lisp :cl-cuda) 3 | (:import-from :neuralisp.core.tensor :tensor :tensor-data :tensor-shape :make-tensor 4 | :tensor-gpu-pointer) 5 | (:export :initialize-gpu :shutdown-gpu :to-gpu :from-gpu :gpu-allocate :gpu-deallocate 6 | :tensor-on-gpu-p :tensor-ensure-on-gpu :tensor-ensure-on-cpu :move-to-gpu :move-to-cpu)) 7 | (in-package :neuralisp.core.gpu) 8 | 9 | (defun initialize-gpu () 10 | "Initialize the GPU and cl-cuda library." 11 | (setf cl-cuda.all:use-cache-p t) ; Enable caching of compiled CUDA programs 12 | (dolist (platform (cl-cuda.all:get-platform-ids)) 13 | (dolist (device (cl-cuda.all:get-device-ids platform)) 14 | (format t "Platform: ~A, Device: ~A~%" platform device)))) 15 | 16 | (defun shutdown-gpu () 17 | "Clean up the GPU and cl-cuda library before exiting." 18 | (cl-cuda.basic:shutdown)) 19 | 20 | (defun to-gpu (tensor) 21 | "Send the tensor to GPU memory." 22 | (let ((tensor-dev-ptr (cublas:allocate (reduce #'* (tensor-shape tensor))))) 23 | (cublas:with-cublas 24 | (cublas:send-to tensor-dev-ptr (tensor-data tensor) (reduce #'* (tensor-shape tensor)))) 25 | tensor-dev-ptr)) 26 | 27 | (defun from-gpu (tensor-dev-ptr shape) 28 | "Retrieve tensor from GPU memory, given its device-pointer and shape." 29 | (let ((tensor-data (make-array (reduce #'* shape) :element-type 'single-float))) 30 | (cublas:with-cublas 31 | (cublas:retrieve-from tensor-dev-ptr tensor-data (reduce #'* shape))) 32 | (make-tensor tensor-data shape))) 33 | 34 | (defmacro gpu-allocate (var &rest args) 35 | "Create and allocate GPU memory for a tensor, storing its device-pointer in the given var." 36 | `(let ((,var (cublas:allocate (reduce #'* ',args)))) 37 | ,var)) 38 | 39 | (defmacro gpu-deallocate (var) 40 | "Deallocate GPU memory associated with a tensor's device-pointer." 41 | `(cublas:deallocate ,var)) 42 | 43 | (defun tensor-on-gpu-p (tensor) 44 | "Verify if the provided tensor is stored on the GPU." 45 | (eql (tensor-data tensor) :gpu)) 46 | 47 | (defun move-to-gpu (tensor) 48 | "Move the provided tensor to the GPU." 49 | (if (tensor-on-gpu-p tensor) 50 | tensor 51 | (make-instance 'tensor :data :gpu 52 | :shape (tensor-shape tensor)))) 53 | 54 | (defun move-to-cpu (tensor) 55 | "Move the provided tensor back to the CPU." 56 | (if (tensor-on-gpu-p tensor) 57 | (make-instance 'tensor :data (copy-seq (tensor-data tensor)) :shape (tensor-shape tensor)) 58 | tensor)) 59 | 60 | (defun tensor-ensure-on-gpu (tensor) 61 | "Ensures tensor's data is on GPU. If not, sends the data to GPU." 62 | (unless (tensor-on-gpu-p tensor) 63 | (setf (tensor-gpu-pointer tensor) (to-gpu tensor)) 64 | (setf (tensor-data tensor) nil)) 65 | tensor) 66 | 67 | (defun tensor-ensure-on-cpu (tensor) 68 | "Ensures tensor's data is on CPU. If not, retrieves the data from GPU." 69 | (when (tensor-on-gpu-p tensor) 70 | (setf (tensor-data tensor) (from-gpu (tensor-gpu-pointer tensor) (tensor-shape tensor))) 71 | (gpu-deallocate (tensor-gpu-pointer tensor)) 72 | (setf (tensor-gpu-pointer tensor) nil)) 73 | tensor) -------------------------------------------------------------------------------- /src/core/tensor.lisp: -------------------------------------------------------------------------------- 1 | (defpackage :neuralisp.core.tensor 2 | (:use :cl) 3 | (:import-from :magicl :matrix :eql :addf :subf :mult :divf 4 | :transpose :dot :sum :mean) 5 | (:export :tensor :tensor-data :tensor-shape :make-tensor 6 | :tensor-add :tensor-subtract :tensor-multiply :tensor-divide 7 | :tensor-matmul :tensor-sum :tensor-mean)) 8 | (in-package :neuralisp.core.tensor) 9 | 10 | (defclass tensor () 11 | ((data :initarg :data 12 | :accessor tensor-data 13 | :type magicl:matrix 14 | :documentation "N-dimensional array holding the tensor's data.") 15 | (shape :initarg :shape 16 | :accessor tensor-shape 17 | :type list 18 | :documentation "List of integers representing the tensor's shape.") 19 | (gpu-pointer :initform nil 20 | :accessor tensor-gpu-pointer 21 | :type (or null cl-cuda.buffer:cublas-device-pointer) 22 | :documentation "Pointer to tensor's data on GPU memory."))) 23 | 24 | (defun make-tensor (shape &key (initial-element 0) (on-gpu nil)) 25 | "Create a new tensor from input 'shape' and initialize it with 'initial-element'. 26 | Optionally move the tensor to GPU memory if 'on-gpu' is true." 27 | (let ((tensor (make-instance 'tensor 28 | :data (magicl:const initial-element shape :layout :row-major) 29 | :shape shape))) 30 | (when on-gpu 31 | (move-to-gpu tensor)) 32 | tensor)) 33 | 34 | ; Basic tensor operations 35 | 36 | (defun tensor-add (tensor-a tensor-b) 37 | "Compute the element-wise addition of two tensors." 38 | (let ((res (magicl:copy-matrix (tensor-data tensor-a)))) 39 | (magicl:addf res (tensor-data tensor-b)) 40 | (make-tensor (tensor-shape tensor-a) :data res))) 41 | 42 | (defun tensor-subtract (tensor-a tensor-b) 43 | "Compute the element-wise subtraction of two tensors." 44 | (let ((res (magicl:copy-matrix (tensor-data tensor-a)))) 45 | (magicl:subf res (tensor-data tensor-b)) 46 | (make-tensor (tensor-shape tensor-a) :data res))) 47 | 48 | (defun tensor-multiply (tensor-a tensor-b) 49 | "Compute the element-wise multiplication of two tensors." 50 | (let ((res (magicl:copy-matrix (tensor-data tensor-a)))) 51 | (magicl:mult ".*" res (tensor-data tensor-b)) 52 | (make-tensor (tensor-shape tensor-a) :data res))) 53 | 54 | (defun tensor-divide (tensor-a tensor-b) 55 | "Compute the element-wise division of two tensors." 56 | (let ((res (magicl:copy-matrix (tensor-data tensor-a)))) 57 | (magicl:divf ".*" res (tensor-data tensor-b)) 58 | (make-tensor (tensor-shape tensor-a) :data res))) 59 | 60 | ; Matrix multiplication and reduction functions 61 | 62 | (defun tensor-matmul (tensor-a tensor-b &key (transpose-a nil) (transpose-b nil)) 63 | "Compute the matrix multiplication of two tensors." 64 | (let* ((a (if transpose-a (magicl:transpose (tensor-data tensor-a)) (tensor-data tensor-a))) 65 | (b (if transpose-b (magicl:transpose (tensor-data tensor-b)) (tensor-data tensor-b))) 66 | (res (magicl:mult a b))) 67 | (make-tensor (list (magicl:matrix-rows res) (magicl:matrix-cols res)) :data res))) 68 | 69 | (defun tensor-sum (tensor &key (axis nil) (keepdims nil)) 70 | "Compute the sum of tensor elements along the specified axis or axes." 71 | (let ((sum (magicl:sum (tensor-data tensor) :axis axis :keepdims keepdims))) 72 | (if keepdims 73 | (make-tensor (magicl:matrix-shape sum) :data sum) 74 | sum))) ; return scalar 75 | 76 | (defun tensor-mean (tensor &key (axis nil) (keepdims nil)) 77 | "Compute the mean of tensor elements along the specified axis or axes." 78 | (let ((mean (magicl:mean (tensor-data tensor) :axis axis :keepdims keepdims))) 79 | (if keepdims 80 | (make-tensor (magicl:matrix-shape mean) :data mean) 81 | mean))) ; return scalar -------------------------------------------------------------------------------- /src/layers/attention.lisp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ck46/neuralisp/b9b594c0e83ea76cbb11598b7f495fbb97ba90ca/src/layers/attention.lisp -------------------------------------------------------------------------------- /src/layers/base.lisp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ck46/neuralisp/b9b594c0e83ea76cbb11598b7f495fbb97ba90ca/src/layers/base.lisp -------------------------------------------------------------------------------- /src/layers/convolutional.lisp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ck46/neuralisp/b9b594c0e83ea76cbb11598b7f495fbb97ba90ca/src/layers/convolutional.lisp -------------------------------------------------------------------------------- /src/layers/linear.lisp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ck46/neuralisp/b9b594c0e83ea76cbb11598b7f495fbb97ba90ca/src/layers/linear.lisp -------------------------------------------------------------------------------- /src/layers/multihead_attention.lisp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ck46/neuralisp/b9b594c0e83ea76cbb11598b7f495fbb97ba90ca/src/layers/multihead_attention.lisp -------------------------------------------------------------------------------- /src/layers/recurrent.lisp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ck46/neuralisp/b9b594c0e83ea76cbb11598b7f495fbb97ba90ca/src/layers/recurrent.lisp -------------------------------------------------------------------------------- /src/losses/base.lisp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ck46/neuralisp/b9b594c0e83ea76cbb11598b7f495fbb97ba90ca/src/losses/base.lisp -------------------------------------------------------------------------------- /src/losses/cross_entropy.lisp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ck46/neuralisp/b9b594c0e83ea76cbb11598b7f495fbb97ba90ca/src/losses/cross_entropy.lisp -------------------------------------------------------------------------------- /src/losses/mse.lisp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ck46/neuralisp/b9b594c0e83ea76cbb11598b7f495fbb97ba90ca/src/losses/mse.lisp -------------------------------------------------------------------------------- /src/optimizers/adam.lisp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ck46/neuralisp/b9b594c0e83ea76cbb11598b7f495fbb97ba90ca/src/optimizers/adam.lisp -------------------------------------------------------------------------------- /src/optimizers/base.lisp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ck46/neuralisp/b9b594c0e83ea76cbb11598b7f495fbb97ba90ca/src/optimizers/base.lisp -------------------------------------------------------------------------------- /src/optimizers/sgd.lisp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ck46/neuralisp/b9b594c0e83ea76cbb11598b7f495fbb97ba90ca/src/optimizers/sgd.lisp -------------------------------------------------------------------------------- /src/transformers/decoder.lisp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ck46/neuralisp/b9b594c0e83ea76cbb11598b7f495fbb97ba90ca/src/transformers/decoder.lisp -------------------------------------------------------------------------------- /src/transformers/encoder.lisp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ck46/neuralisp/b9b594c0e83ea76cbb11598b7f495fbb97ba90ca/src/transformers/encoder.lisp -------------------------------------------------------------------------------- /src/transformers/transformer_layer.lisp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ck46/neuralisp/b9b594c0e83ea76cbb11598b7f495fbb97ba90ca/src/transformers/transformer_layer.lisp -------------------------------------------------------------------------------- /src/utils/data_loader.lisp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ck46/neuralisp/b9b594c0e83ea76cbb11598b7f495fbb97ba90ca/src/utils/data_loader.lisp -------------------------------------------------------------------------------- /src/utils/debugging.lisp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ck46/neuralisp/b9b594c0e83ea76cbb11598b7f495fbb97ba90ca/src/utils/debugging.lisp -------------------------------------------------------------------------------- /tests/activations/test_activations.lisp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ck46/neuralisp/b9b594c0e83ea76cbb11598b7f495fbb97ba90ca/tests/activations/test_activations.lisp -------------------------------------------------------------------------------- /tests/core/test_autograd.lisp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ck46/neuralisp/b9b594c0e83ea76cbb11598b7f495fbb97ba90ca/tests/core/test_autograd.lisp -------------------------------------------------------------------------------- /tests/core/test_gpu.lisp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ck46/neuralisp/b9b594c0e83ea76cbb11598b7f495fbb97ba90ca/tests/core/test_gpu.lisp -------------------------------------------------------------------------------- /tests/core/test_tensor.lisp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ck46/neuralisp/b9b594c0e83ea76cbb11598b7f495fbb97ba90ca/tests/core/test_tensor.lisp -------------------------------------------------------------------------------- /tests/layers/test_layers.lisp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ck46/neuralisp/b9b594c0e83ea76cbb11598b7f495fbb97ba90ca/tests/layers/test_layers.lisp -------------------------------------------------------------------------------- /tests/losses/test_losses.lisp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ck46/neuralisp/b9b594c0e83ea76cbb11598b7f495fbb97ba90ca/tests/losses/test_losses.lisp -------------------------------------------------------------------------------- /tests/optimizers/test_optimizers.lisp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ck46/neuralisp/b9b594c0e83ea76cbb11598b7f495fbb97ba90ca/tests/optimizers/test_optimizers.lisp -------------------------------------------------------------------------------- /tests/transformers/test_transformers.lisp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ck46/neuralisp/b9b594c0e83ea76cbb11598b7f495fbb97ba90ca/tests/transformers/test_transformers.lisp --------------------------------------------------------------------------------