├── .github └── ISSUE_TEMPLATE │ └── bug_report.md ├── .gitignore ├── .gitmodules ├── CITATION.cff ├── LICENSE ├── README.md ├── asr-version.txt ├── asr_calibrate.m ├── asr_calibrate_r.m ├── asr_process.m ├── asr_process_r.m ├── clean_artifacts.m ├── clean_asr.m ├── clean_channels.m ├── clean_channels_nolocs.m ├── clean_drifts.m ├── clean_flatlines.m ├── clean_rawdata.m ├── clean_windows.m ├── eegplugin_clean_rawdata.m ├── gui_interface.ai ├── gui_interface.png ├── pop_clean_rawdata.m ├── private ├── block_geometric_median.m ├── design_fir.m ├── design_kaiser.m ├── filter_fast.m ├── filtfilt_fast.m ├── findjobj.m ├── fit_eeg_distribution.m ├── geometric_median.m ├── hlp_handleerror.m ├── hlp_microcache.m ├── hlp_split.m ├── hlp_varargin2struct.m ├── sphericalSplineInterpolate.m └── window_func.m ├── rasr_nonlinear_eigenspace.m ├── readme-asr.txt ├── readme-extras.txt └── vis_artifacts.m /.github/ISSUE_TEMPLATE/bug_report.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Bug report 3 | about: Create a report to help us improve 4 | title: '' 5 | labels: '' 6 | assignees: '' 7 | 8 | --- 9 | 10 | Please, do not submit bugs here. 11 | 12 | For faster processing, please submit bugs at https://github.com/sccn/eeglab/issues 13 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | *.asv 2 | -------------------------------------------------------------------------------- /.gitmodules: -------------------------------------------------------------------------------- 1 | [submodule "manopt"] 2 | path = manopt 3 | url = https://github.com/NicolasBoumal/manopt.git 4 | -------------------------------------------------------------------------------- /CITATION.cff: -------------------------------------------------------------------------------- 1 | cff-version: 1.2.0 2 | message: "If you use this software, please cite it as below." 3 | authors: 4 | - family-names: Kothe, Miyakoshi, Delorme 5 | given-names: Christian, Makoto, Arnaud 6 | orcid: https://orcid.org/0000-0002-4062-6679, https://orcid.org/0000-0002-0799-3557 7 | title: "clean_rawdata" 8 | version: 2.7 9 | date-released: 2019-01-19 10 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Clean_rawdata EEGLAB plug-in 2 | 3 | The Clean Rawdata plug-in (version 2.0) interface has been redesigned and will soon become the default EEGLAB method for removing artifacts from EEG and related data. The plug-in detects and can separate low-frequency drifts, flatline and noisy channels from the data. It can also apply ASR (automated subspace removal) to detect and reject or remove high-amplitude non-brain ('artifact') activity (produced by eye blinks, muscle activity, sensor motion, etc.) by comparing its structure to that of known artifact-free reference data, thereby revealing and recovering (possibly smaller) EEG background activity that lies outside the subspace spanned by the artifact processes. 4 | 5 | **Note:** This plug-in uses the Signal Processing toolbox for pre- and post-processing of the data (removing drifts, channels and time windows); the core ASR method (clean_asr) does not require this toolbox but you will need high-pass filtered data if you use it directly. 6 | 7 | # This project needs you 8 | 9 | We need community maintain to this project. Please review existing issues and issue pull requests. A section in this documentation with link to all the existing methodological papers is also needed. 10 | 11 | # Credit 12 | 13 | This plug-in, clean_rawdata uses methods (e.g., Artifact Subspace 14 | Reconstruction, ASR) by Christian Kothe from the BCILAB Toolbox 15 | (Kothe & Makeig, 2013), first wrapped into an EEGLAB plug-in by 16 | Makoto Miyakoshi and further developed by Arnaud Delorme with 17 | Scott Makeig. 18 | 19 | This plug-in cleans raw EEG data. Methods from the BCILAB toolbox 20 | are being used (in particular Artifact Subspace Reconstruction) 21 | designed by Christian Kothe. 22 | 23 | These functions were wrapped up into an EEGLAB plug-in by Makoto 24 | Myakoshi, then later by Arnaud Delorme with input from Scott 25 | Makeig. 26 | 27 | The private folder contains 3rd party utilities, including: 28 | - findjobj.m Copyright (C) 2007-2010 Yair M. Altman 29 | - asr_calibrate.m and asr_process.m 30 | Copyright (C) 2013 The Regents of the University of California 31 | Note that this function is not free for commercial use. 32 | - sperhicalSplineInterpolate.m Copyright (C) 2009 Jason Farquhar 33 | - oct_fftfilt Copyright (C) 1996, 1997 John W. Eaton 34 | - utility functions from the BCILAB toolbox Copyright (C) 2010-2014 Christian Kothe 35 | 36 | The folder "manopt" contains the Matlab toolbox for optimization on manifolds. 37 | 38 | # Graphic interface 39 | 40 | Below we detail the GUI interface. Individual function contain additional help information. 41 | 42 | ![](gui_interface.png) 43 | 44 | ## High pass filter the data 45 | 46 | Check checkbox **(1)** if the data have not been high pass filtered yet. If you use this option, the edit box in **(2)** allows setting the transition band for the high-pass filter in Hz. This is formatted as[transition-start, transition-end]. Default is 0.25 to 0.75 Hz. 47 | 48 | ## Reject bad channels 49 | 50 | Check checkbox **(3)** to reject bad channels. Options **(4)** allows removal of flat channels. The edit box sets the maximum tolerated (non-rejected) flatline duration in seconds. If a channel has a longer flatline than this, it will be considered abnormal and rejected. The default is 5 seconds. Option **(5)** sets the Line Noise criterion: If a channel has more line noise relative to its signal than this value (in standard deviations based on the total channel signal), it is considered abnormal. The default is 4 standard deviations. Option **(6)** sets the minimum channel correlation. If a channel is correlated at less than this value to an estimate based on other nearby channels, it is considered abnormal in the given time window. This method requires that channel locations be available and roughly correct; otherwise a fallback criterion will be used. The default is a correlation of 0.8. 51 | 52 | ## Artifact Subspace Reconstruction 53 | 54 | Check checkbox **(7)** to use Artifact Subspace Reconstruction (ASR). ASR is described in this [publication](https://www.ncbi.nlm.nih.gov/pubmed/26415149). In edit box **(8)** you may change the standard deviation cutoff for removal of bursts (via ASR). Data portions whose variance is larger than this threshold relative to the calibration data are considered missing data and will be removed. The most aggressive value that can be used without losing much EEG is 3. For new users it is recommended to first visually inspect the difference between the aw and the cleaned data (using eegplot) to get a sense of the content the is removed at various levels of this input variable. Here, a quite conservative value is 20; this is the current default value. Use edit box **(9)** to use Riemannian distance instead of Euclidian distance. This is a beta option as the advantage of this method has not yet been clearly demonstrated. Checkbox **(10)** allows removal instead of correction of artifact-laden portions of data identified by ASR. One of the strength of ASR is its ability to detect stretches of 'bad data' before correcting them. This option allows use of ASR for data-period rejection instead of correction, and is the default for offline data processing. ASR was originally designed as an online data cleaning algorithm, in which case 'bad data' correction may be used. 55 | 56 | ## Additional removal of 'bad data' periods 57 | 58 | Check checkbox **(11)** to perform additional removal of bad-data periods. Edit box **(12)** sets the maximum percentage of contaminated channels that are tolerated in the final output data for each considered window. Edit box **(13)** sets the noise threshold for labeling a channel as contaminated. 59 | 60 | ## Display rejected and corrected regions 61 | 62 | Check checkbox **(14)** plots rejection results overlaid on the original data. This option is useful to visually assess the performance of a given ASR method. 63 | 64 | Additional parameters are accessible through the command line interface of the clean_artifacts function. 65 | 66 | ## Additional documentation 67 | 68 | Makoto Miyakoshi wrote a page in the [wiki section](https://github.com/sccn/clean_rawdata/wiki) of this repository discussing ASR. 69 | 70 | # Version history 71 | v0.34 and earlier - original versions 72 | 73 | v1.0 - new default values for some of the rejection tools, new GUI 74 | 75 | v2.0 - new improved GUI, compatibility with studies 76 | 77 | v2.1 - fix issue with 'distance' variable for burst detection 78 | 79 | v2.2 - fix history call for pop_clean_rawdata 80 | 81 | v2.3 - add maxmem to asr_calibrate to ensure reproducibility of results 82 | 83 | v2.4 - fixing issue with running function in parallel for Matlab 2020a 84 | 85 | v2.5 - move asr_calibrate out of the private folder so it can be used directly 86 | 87 | v2.6 - allowing to exclude channels and a variety of small bug fixes 88 | 89 | v2.7 - allowing to fuse channel rejection for datasets with same subject and session (STUDY processing) 90 | 91 | v2.8 - better error messages, and fix excluding channels (there was a rare crash) 92 | 93 | v2.9 - fix bug when ignoring channels and removing channels at the same time, fix plotting issue with vis_artifact 94 | 95 | v2.9.1 - add support for fractional sampling rate; fix too many splits with high sampling frequencies 96 | 97 | v2.10 - The code was updated to avoid crashes and fix minor errors related to channel masking, array dimensions, and ignoring channels. 98 | -------------------------------------------------------------------------------- /asr-version.txt: -------------------------------------------------------------------------------- 1 | 0.13 2 | 3 | Changes since version 0.12: 4 | - removed signal processing and statistics toolbox dependency 5 | - now using a more robust threshold estimation procedure in clean_windows and clean_asr 6 | - vis_artifacts now properly displays events while scrolling through data 7 | - fix: in asr_process when the reconstruction matrix was nontrivial on the first sample (would crash) 8 | 9 | Changes since version 0.1: 10 | - added proper licenses to code files 11 | - during online processing asr_process.m now carries over the reconstruction matrix from the previous block 12 | - minor fix: the clean_asr.m function now returns the same number of samples as its input 13 | 14 | -------------------------------------------------------------------------------- /asr_process.m: -------------------------------------------------------------------------------- 1 | function [outdata,outstate] = asr_process(data,srate,state,windowlen,lookahead,stepsize,maxdims,maxmem,usegpu) 2 | % Processing function for the Artifact Subspace Reconstruction (ASR) method. 3 | % [Data,State] = asr_process(Data,SamplingRate,State,WindowLength,LookAhead,StepSize,MaxDimensions,MaxMemory,UseGPU) 4 | % 5 | % This function is used to clean multi-channel signal using the ASR method. The required inputs are 6 | % the data matrix, the sampling rate of the data, and the filter state (as initialized by 7 | % asr_calibrate). If the data is used on successive chunks of data, the output state of the previous 8 | % call to asr_process should be passed in. 9 | % 10 | % In: 11 | % Data : Chunk of data to process [#channels x #samples]. This is a chunk of data, assumed to be 12 | % a continuation of the data that was passed in during the last call to asr_process (if 13 | % any). The data should be *zero-mean* (e.g., high-pass filtered the same way as for 14 | % asr_calibrate). 15 | % 16 | % SamplingRate : sampling rate of the data in Hz (e.g., 250.0) 17 | % 18 | % State : initial filter state (determined by asr_calibrate or from previous call to asr_process) 19 | % 20 | % WindowLength : Length of the statistcs window, in seconds (e.g., 0.5). This should not be much 21 | % longer than the time scale over which artifacts persist, but the number of samples 22 | % in the window should not be smaller than 1.5x the number of channels. Default: 0.5 23 | % 24 | % LookAhead : Amount of look-ahead that the algorithm should use. Since the processing is causal, 25 | % the output signal will be delayed by this amount. This value is in seconds and should 26 | % be between 0 (no lookahead) and WindowLength/2 (optimal lookahead). The recommended 27 | % value is WindowLength/2. Default: WindowLength/2 28 | % 29 | % StepSize : The statistics will be updated every this many samples. The larger this is, the faster 30 | % the algorithm will be. The value must not be larger than WindowLength*SamplingRate. 31 | % The minimum value is 1 (update for every sample) while a good value is 1/3 of a second. 32 | % Note that an update is always performed also on the first and last sample of the data 33 | % chunk. Default: 32 34 | % 35 | % MaxDimensions : Maximum dimensionality of artifacts to remove. Up to this many dimensions (or up 36 | % to this fraction of dimensions) can be removed for a given data segment. If the 37 | % algorithm needs to tolerate extreme artifacts a higher value than the default 38 | % may be used (the maximum fraction is 1.0). Default 0.66 39 | % 40 | % MaxMemory : The maximum amount of memory used by the algorithm when processing a long chunk with 41 | % many channels, in MB. The recommended value is at least 64. To run on the GPU, use 42 | % the amount of memory available to your GPU here (needs the parallel computing toolbox). 43 | % default: min(5000,1/2 * free memory in MB). Using smaller amounts of memory leads to 44 | % longer running times. 45 | % 46 | % UseGPU : Whether to run on the GPU. This makes sense for offline processing if you have a a card 47 | % with enough memory and good double-precision performance (e.g., NVIDIA GTX Titan or 48 | % K20). Note that for this to work you need to have the Parallel Computing toolbox. 49 | % Default: false 50 | % 51 | % Out: 52 | % Data : cleaned data chunk (same length as input but delayed by LookAhead samples) 53 | % 54 | % State : final filter state (can be passed in for subsequent calls) 55 | % 56 | 57 | % History 58 | % 03/20/2019 Makoto and Chiyuan. Supported 'availableRAM_GB'. GUI switched to GUIDE-made. 59 | % 08/31/2012 Christian. Created. 60 | % 61 | % Christian Kothe, Swartz Center for Computational Neuroscience, UCSD 62 | % 2012-08-31 63 | 64 | % UC Copyright Notice 65 | % This software is Copyright (C) 2013 The Regents of the University of California. All Rights Reserved. 66 | % 67 | % Permission to copy, modify, and distribute this software and its documentation for educational, 68 | % research and non-profit purposes, without fee, and without a written agreement is hereby granted, 69 | % provided that the above copyright notice, this paragraph and the following three paragraphs appear 70 | % in all copies. 71 | % 72 | % Permission to make commercial use of this software may be obtained by contacting: 73 | % Technology Transfer Office 74 | % 9500 Gilman Drive, Mail Code 0910 75 | % University of California 76 | % La Jolla, CA 92093-0910 77 | % (858) 534-5815 78 | % invent@ucsd.edu 79 | % 80 | % This software program and documentation are copyrighted by The Regents of the University of 81 | % California. The software program and documentation are supplied "as is", without any accompanying 82 | % services from The Regents. The Regents does not warrant that the operation of the program will be 83 | % uninterrupted or error-free. The end-user understands that the program was developed for research 84 | % purposes and is advised not to rely exclusively on the program for any reason. 85 | % 86 | % IN NO EVENT SHALL THE UNIVERSITY OF CALIFORNIA BE LIABLE TO ANY PARTY FOR DIRECT, INDIRECT, 87 | % SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES, INCLUDING LOST PROFITS, ARISING OUT OF THE USE OF 88 | % THIS SOFTWARE AND ITS DOCUMENTATION, EVEN IF THE UNIVERSITY OF CALIFORNIA HAS BEEN ADVISED OF THE 89 | % POSSIBILITY OF SUCH DAMAGE. THE UNIVERSITY OF CALIFORNIA SPECIFICALLY DISCLAIMS ANY WARRANTIES, 90 | % INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A 91 | % PARTICULAR PURPOSE. THE SOFTWARE PROVIDED HEREUNDER IS ON AN "AS IS" BASIS, AND THE UNIVERSITY OF 92 | % CALIFORNIA HAS NO OBLIGATIONS TO PROVIDE MAINTENANCE, SUPPORT, UPDATES, ENHANCEMENTS, OR 93 | % MODIFICATIONS. 94 | 95 | if nargin < 4 || isempty(windowlen) 96 | windowlen = 0.5; end 97 | windowlen = max(windowlen,1.5*size(data,1)/srate); 98 | if nargin < 5 || isempty(lookahead) 99 | lookahead = windowlen/2; end 100 | if nargin < 6 || isempty(stepsize) 101 | stepsize = 32; end 102 | if nargin < 7 || isempty(maxdims) 103 | maxdims = 0.66; end 104 | if nargin < 9 || isempty(usegpu) 105 | usegpu = false; end 106 | if nargin < 8 || isempty(maxmem) 107 | if usegpu 108 | dev = gpuDevice(); maxmem = dev.FreeMemory/2^20; 109 | else 110 | maxmem = hlp_memfree/(2^21); % In MB? (if 2^20) Probably because in the subsequent moving average process, you'll need to secure double amount of data. 111 | end 112 | end 113 | if maxdims < 1 114 | maxdims = round(size(data,1)*maxdims); end 115 | if isempty(data) 116 | outdata = data; outstate = state; return; end 117 | 118 | [C,S] = size(data); 119 | N = round(windowlen*srate); 120 | P = round(lookahead*srate); 121 | [T,M,A,B] = deal(state.T,state.M,state.A,state.B); % T, threshold; M, mixing matrix from the calibration data (sqrt of covariance matrix), IIR filter, IIR filter 122 | 123 | % initialize prior filter state by extrapolating available data into the past (if necessary) 124 | if isempty(state.carry) 125 | state.carry = repmat(2*data(:,1),1,P) - data(:,1+mod(((P+1):-1:2)-1,S)); end 126 | 127 | data = [state.carry data]; 128 | data(~isfinite(data(:))) = 0; 129 | 130 | % split up the total sample range into k chunks that will fit in memory 131 | if maxmem*1024*1024 - C*C*P*8*3 < 0 132 | disp('Memory too low, increasing it (rejection block size now depends on available memory so it might not be 100% reproducible)...'); 133 | maxmem = hlp_memfree/(2^21); 134 | if maxmem*1024*1024 - C*C*P*8*3 < 0 135 | error('Not enough memory'); 136 | end 137 | end 138 | splits = ceil((C*C*S*8*8 + C*C*8*S/stepsize + C*S*8*2 + S*8*5) / (maxmem*1024*1024 - C*C*P*8*3)); % Mysterious. More memory available, less 'splits' 139 | % there is something wrong with the formula above (1.6M splits for 140 | % subject 2252A of ds003947; capping at 10000 below) 141 | splits = min(splits, 10000); 142 | if splits > 1 143 | fprintf('Now cleaning data in %i blocks',splits); end 144 | 145 | for i=1:splits 146 | range = 1+floor((i-1)*S/splits) : min(S,floor(i*S/splits)); % 147 | if ~isempty(range) 148 | % get spectrally shaped data X for statistics computation (range shifted by lookahead) 149 | [X,state.iir] = filter(B,A,double(data(:,range+P)),state.iir,2); 150 | % move it to the GPU if applicable 151 | if usegpu && length(range) > 1000 152 | try X = gpuArray(X); catch,end; end 153 | % compute running mean covariance (assuming a zero-mean signal) 154 | [Xcov,state.cov] = moving_average(N,reshape(bsxfun(@times,reshape(X,1,C,[]),reshape(X,C,1,[])),C*C,[]),state.cov); % ch c ch x range. 155 | % extract the subset of time points at which we intend to update 156 | update_at = min(stepsize:stepsize:(size(Xcov,2)+stepsize-1),size(Xcov,2)); 157 | % if there is no previous R (from the end of the last chunk), we estimate it right at the first sample 158 | if isempty(state.last_R) 159 | update_at = [1 update_at]; 160 | state.last_R = eye(C); 161 | end 162 | Xcov = reshape(Xcov(:,update_at),C,C,[]); 163 | if usegpu 164 | Xcov = gather(Xcov); end 165 | % do the reconstruction in intervals of length stepsize (or shorter if at the end of a chunk) 166 | last_n = 0; 167 | for j=1:length(update_at) 168 | % do a PCA to find potential artifact components 169 | [V,D] = eig(Xcov(:,:,j)); 170 | [D,order] = sort(reshape(diag(D),1,C)); V = V(:,order); 171 | % determine which components to keep (variance below directional threshold or not admissible for rejection) 172 | keep = D 1 191 | fprintf('.'); end 192 | end 193 | if splits > 1 194 | fprintf('\n'); end 195 | 196 | % carry the look-ahead portion of the data over to the state (for successive calls) 197 | state.carry = [state.carry data(:,(end-P+1):end)]; 198 | state.carry = state.carry(:,(end-P+1):end); 199 | 200 | % finalize outputs 201 | outdata = data(:,1:(end-P)); 202 | if usegpu 203 | state.iir = gather(state.iir); 204 | state.cov = gather(state.cov); 205 | end 206 | outstate = state; 207 | 208 | 209 | 210 | function [X,Zf] = moving_average(N,X,Zi) 211 | % Run a moving-average filter along the second dimension of the data. 212 | % [X,Zf] = moving_average(N,X,Zi) 213 | % 214 | % In: 215 | % N : filter length in samples 216 | % X : data matrix [#Channels x #Samples] 217 | % Zi : initial filter conditions (default: []) 218 | % 219 | % Out: 220 | % X : the filtered data 221 | % Zf : final filter conditions 222 | % 223 | % Christian Kothe, Swartz Center for Computational Neuroscience, UCSD 224 | % 2012-01-10 225 | 226 | if nargin <= 2 || isempty(Zi) 227 | Zi = zeros(size(X,1),N); end 228 | 229 | % pre-pend initial state & get dimensions 230 | Y = [Zi X]; M = size(Y,2); 231 | % get alternating index vector (for additions & subtractions) 232 | I = [1:M-N; 1+N:M]; 233 | % get sign vector (also alternating, and includes the scaling) 234 | S = [-ones(1,M-N); ones(1,M-N)]/N; 235 | % run moving average 236 | X = cumsum(bsxfun(@times,Y(:,I(:)),S(:)'),2); 237 | % read out result 238 | X = X(:,2:2:end); 239 | 240 | if nargout > 1 241 | Zf = [-(X(:,end)*N-Y(:,end-N+1)) Y(:,end-N+2:end)]; end 242 | 243 | 244 | 245 | function result = hlp_memfree 246 | % Get the amount of free physical memory, in bytes 247 | result = javaMethod('getFreePhysicalMemorySize', javaMethod('getOperatingSystemMXBean','java.lang.management.ManagementFactory')); 248 | -------------------------------------------------------------------------------- /asr_process_r.m: -------------------------------------------------------------------------------- 1 | function [outdata,outstate, Y] = asr_process(data,srate,state,windowlen,lookahead,stepsize,maxdims,maxmem,usegpu) 2 | % Processing function for the Artifact Subspace Reconstruction (ASR) method. 3 | % [Data,State] = asr_process(Data,SamplingRate,State,WindowLength,LookAhead,StepSize,MaxDimensions,MaxMemory,UseGPU) 4 | % 5 | % This function is used to clean multi-channel signal using the ASR method. The required inputs are 6 | % the data matrix, the sampling rate of the data, and the filter state (as initialized by 7 | % asr_calibrate). If the data is used on successive chunks of data, the output state of the previous 8 | % call to asr_process should be passed in. 9 | % 10 | % In: 11 | % Data : Chunk of data to process [#channels x #samples]. This is a chunk of data, assumed to be 12 | % a continuation of the data that was passed in during the last call to asr_process (if 13 | % any). The data should be *zero-mean* (e.g., high-pass filtered the same way as for 14 | % asr_calibrate). 15 | % 16 | % SamplingRate : sampling rate of the data in Hz (e.g., 250.0) 17 | % 18 | % State : initial filter state (determined by asr_calibrate or from previous call to asr_process) 19 | % 20 | % WindowLength : Length of the statistcs window, in seconds (e.g., 0.5). This should not be much 21 | % longer than the time scale over which artifacts persist, but the number of samples 22 | % in the window should not be smaller than 1.5x the number of channels. Default: 0.5 23 | % 24 | % LookAhead : Amount of look-ahead that the algorithm should use. Since the processing is causal, 25 | % the output signal will be delayed by this amount. This value is in seconds and should 26 | % be between 0 (no lookahead) and WindowLength/2 (optimal lookahead). The recommended 27 | % value is WindowLength/2. Default: WindowLength/2 28 | % 29 | % StepSize : The statistics will be updated every this many samples. The larger this is, the faster 30 | % the algorithm will be. The value must not be larger than WindowLength*SamplingRate. 31 | % The minimum value is 1 (update for every sample) while a good value is 1/3 of a second. 32 | % Note that an update is always performed also on the first and last sample of the data 33 | % chunk. Default: 32 34 | % 35 | % MaxDimensions : Maximum dimensionality of artifacts to remove. Up to this many dimensions (or up 36 | % to this fraction of dimensions) can be removed for a given data segment. If the 37 | % algorithm needs to tolerate extreme artifacts a higher value than the default 38 | % may be used (the maximum fraction is 1.0). Default 0.66 39 | % 40 | % MaxMemory : The maximum amount of memory used by the algorithm when processing a long chunk with 41 | % many channels, in MB. The recommended value is at least 64. To run on the GPU, use 42 | % the amount of memory available to your GPU here (needs the parallel computing toolbox). 43 | % default: min(5000,1/2 * free memory in MB). Using smaller amounts of memory leads to 44 | % longer running times. 45 | % 46 | % UseGPU : Whether to run on the GPU. This makes sense for offline processing if you have a a card 47 | % with enough memory and good double-precision performance (e.g., NVIDIA GTX Titan or 48 | % K20). Note that for this to work you need to have the Parallel Computing toolbox. 49 | % Default: false 50 | % 51 | % Out: 52 | % Data : cleaned data chunk (same length as input but delayed by LookAhead samples) 53 | % 54 | % State : final filter state (can be passed in for subsequent calls) 55 | % 56 | % Christian Kothe, Swartz Center for Computational Neuroscience, UCSD 57 | % 2012-08-31 58 | 59 | % UC Copyright Notice 60 | % This software is Copyright (C) 2013 The Regents of the University of California. All Rights Reserved. 61 | % 62 | % Permission to copy, modify, and distribute this software and its documentation for educational, 63 | % research and non-profit purposes, without fee, and without a written agreement is hereby granted, 64 | % provided that the above copyright notice, this paragraph and the following three paragraphs appear 65 | % in all copies. 66 | % 67 | % Permission to make commercial use of this software may be obtained by contacting: 68 | % Technology Transfer Office 69 | % 9500 Gilman Drive, Mail Code 0910 70 | % University of California 71 | % La Jolla, CA 92093-0910 72 | % (858) 534-5815 73 | % invent@ucsd.edu 74 | % 75 | % This software program and documentation are copyrighted by The Regents of the University of 76 | % California. The software program and documentation are supplied "as is", without any accompanying 77 | % services from The Regents. The Regents does not warrant that the operation of the program will be 78 | % uninterrupted or error-free. The end-user understands that the program was developed for research 79 | % purposes and is advised not to rely exclusively on the program for any reason. 80 | % 81 | % IN NO EVENT SHALL THE UNIVERSITY OF CALIFORNIA BE LIABLE TO ANY PARTY FOR DIRECT, INDIRECT, 82 | % SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES, INCLUDING LOST PROFITS, ARISING OUT OF THE USE OF 83 | % THIS SOFTWARE AND ITS DOCUMENTATION, EVEN IF THE UNIVERSITY OF CALIFORNIA HAS BEEN ADVISED OF THE 84 | % POSSIBILITY OF SUCH DAMAGE. THE UNIVERSITY OF CALIFORNIA SPECIFICALLY DISCLAIMS ANY WARRANTIES, 85 | % INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A 86 | % PARTICULAR PURPOSE. THE SOFTWARE PROVIDED HEREUNDER IS ON AN "AS IS" BASIS, AND THE UNIVERSITY OF 87 | % CALIFORNIA HAS NO OBLIGATIONS TO PROVIDE MAINTENANCE, SUPPORT, UPDATES, ENHANCEMENTS, OR 88 | % MODIFICATIONS. 89 | 90 | 91 | % Adapted by Sarah Blum, 2018. 92 | % Riemannian processing was added to the processing function in the estimation of the covariance matrix and 93 | % the averaging of covariance matrices. For more details please refer to the paper Blum et al. 2018 (in 94 | % preparation). 95 | 96 | disp('THIS IS RIEMANN ADAPTED PROCESSING!!!'); 97 | if nargin < 4 || isempty(windowlen) 98 | windowlen = 0.1; end 99 | windowlen = max(windowlen,1.5*size(data,1)/srate); 100 | if nargin < 5 || isempty(lookahead) 101 | lookahead = windowlen/2; end 102 | if nargin < 6 || isempty(stepsize) 103 | stepsize = 4; end 104 | if nargin < 7 || isempty(maxdims) 105 | maxdims = 1; end 106 | if nargin < 9 || isempty(usegpu) 107 | usegpu = false; end 108 | if nargin < 8 || isempty(maxmem) 109 | if usegpu 110 | dev = gpuDevice(); maxmem = dev.FreeMemory/2^20; 111 | else 112 | maxmem = hlp_memfree/(2^21); 113 | end 114 | end 115 | if maxdims < 1 116 | maxdims = round(size(data,1)*maxdims); end 117 | if isempty(data) 118 | outdata = data; outstate = state; return; end 119 | 120 | [C,S] = size(data); 121 | N = round(windowlen*srate); 122 | P = round(lookahead*srate); 123 | [T,M,A,B] = deal(state.T,state.M,state.A,state.B); 124 | 125 | % initialize prior filter state by extrapolating available data into the past (if necessary) 126 | if isempty(state.carry) 127 | state.carry = repmat(2*data(:,1),1,P) - data(:,1+mod(((P+1):-1:2)-1,S)); end 128 | 129 | data = [state.carry data]; 130 | data(~isfinite(data(:))) = 0; 131 | 132 | % split up the total sample range into k chunks that will fit in memory 133 | if maxmem*1024*1024 - C*C*P*8*3 < 0 134 | error('Not enough memory'); 135 | end 136 | splits = ceil((C*C*S*8*8 + C*C*8*S/stepsize + C*S*8*2 + S*8*5) / (maxmem*1024*1024 - C*C*P*8*3)); 137 | splits = min(splits, 10000); 138 | if splits > 1 139 | fprintf('Now cleaning data in %i blocks',splits); end 140 | 141 | for i=1:splits 142 | range = 1+floor((i-1)*S/splits) : min(S,floor(i*S/splits)); 143 | if ~isempty(range) 144 | % get spectrally shaped data X for statistics computation (range shifted by lookahead) 145 | % and also get a subrange of the data (according to splits) 146 | [X,state.iir] = filter(B,A,double(data(:,range+P)),state.iir,2); 147 | % return the filtered but othrerwise unaltered data for debugging 148 | Y = X; 149 | % move it to the GPU if applicable 150 | if usegpu && length(range) > 1000 151 | try X = gpuArray(X); catch,end; end 152 | 153 | %% the Riemann version uses the sample covariance matrix here: 154 | SCM = (1/S) * (X * X'); % channels x channels 155 | % if we have a previous covariance matrix, use it to compute the average to make 156 | % the current covariance matrix more stable 157 | if ~isempty(state.cov) 158 | A = zeros([C,C,2]); 159 | A(:,:,1) = SCM; 160 | A(:,:,2) = state.cov; 161 | Xcov = positive_definite_karcher_mean(A); % from Manopt toolbox 162 | else 163 | % we do not have a previous matrix to average, we use SCM as is 164 | Xcov = SCM; 165 | end 166 | 167 | update_at = min(stepsize:stepsize:(size(X,2)+stepsize-1),size(X,2)); 168 | % if there is no previous R (from the end of the last chunk), we estimate it right at the first sample 169 | if isempty(state.last_R) 170 | update_at = [1 update_at]; 171 | state.last_R = eye(C); 172 | end 173 | 174 | % function from manopt toolbox, adapted to this use case. manopt needs to be in the path 175 | %[V1,D1] = eig(Xcov) 176 | [V, D] = rasr_nonlinear_eigenspace(Xcov, C); 177 | % use eigenvalues in descending order 178 | [D, order] = sort(reshape(diag(D),1,C)); 179 | % to sort the eigenvectors, here the vectors computed on the manifold 180 | V = V(:,order); 181 | 182 | % determine which components to keep (variance below directional threshold or not admissible for rejection) 183 | keep = D < sum((T*V).^2) | (1:C)<(C-maxdims); 184 | trivial = all(keep); 185 | 186 | % update the reconstruction matrix R (reconstruct artifact components using the mixing matrix) 187 | if ~trivial 188 | R = real(M*pinv(bsxfun(@times,keep',V'*M))*V'); 189 | else 190 | R = eye(C); 191 | end 192 | 193 | % do the reconstruction in intervals of length stepsize (or shorter at the end of a chunk) 194 | last_n = 0; 195 | for j=1:length(update_at) 196 | % apply the reconstruction to intermediate samples (using raised-cosine blending) 197 | n = update_at(j); 198 | if ~trivial || ~state.last_trivial 199 | subrange = range((last_n+1):n); 200 | blend = (1-cos(pi*(1:(n-last_n))/(n-last_n)))/2; 201 | data(:,subrange) = bsxfun(@times,blend,R*data(:,subrange)) + bsxfun(@times,1-blend,state.last_R*data(:,subrange)); 202 | end 203 | [last_n,state.last_R,state.last_trivial] = deal(n,R,trivial); 204 | end 205 | end 206 | if splits > 1 207 | fprintf('.'); end 208 | end 209 | if splits > 1 210 | fprintf('\n'); end 211 | 212 | % carry the look-ahead portion of the data over to the state (for successive calls) 213 | state.carry = [state.carry data(:,(end-P+1):end)]; 214 | state.carry = state.carry(:,(end-P+1):end); 215 | state.cov = Xcov; 216 | 217 | % finalize outputs 218 | outdata = data(:,1:(end-P)); 219 | outstate = state; 220 | 221 | 222 | 223 | function result = hlp_memfree 224 | % Get the amount of free physical memory, in bytes 225 | result = java.lang.management.ManagementFactory.getOperatingSystemMXBean().getFreePhysicalMemorySize(); 226 | -------------------------------------------------------------------------------- /clean_artifacts.m: -------------------------------------------------------------------------------- 1 | function [EEG,HP,BUR,removed_channels] = clean_artifacts(EEG,varargin) 2 | % All-in-one function for artifact removal, including ASR. 3 | % [EEG,HP,BUR] = clean_artifacts(EEG, Options...) 4 | % 5 | % This function removes flatline channels, low-frequency drifts, noisy channels, short-time bursts 6 | % and incompletely repaird segments from the data. Tip: Any of the core parameters can also be 7 | % passed in as [] to use the respective default of the underlying functions, or as 'off' to disable 8 | % it entirely. 9 | % 10 | % Hopefully parameter tuning should be the exception when using this function -- however, there are 11 | % 3 parameters governing how aggressively bad channels, bursts, and irrecoverable time windows are 12 | % being removed, plus several detail parameters that only need tuning under special circumstances. 13 | % 14 | % Notes: 15 | % * This function uses the Signal Processing toolbox for pre- and post-processing of the data 16 | % (removing drifts, channels and time windows); the core ASR method (clean_asr) does not require 17 | % this toolbox but you will need high-pass filtered data if you use it directly. 18 | % * By default this function will identify subsets of clean data from the given recording to 19 | % enhance the robustness of the ASR calibration phase to strongly contaminated data; this uses 20 | % the Statistics toolbox, but can be skipped/bypassed if needed (see documentation). 21 | % 22 | % In: 23 | % EEG : Raw continuous EEG recording to clean up (as EEGLAB dataset structure). 24 | % 25 | % 26 | % NOTE: The following parameters are the core parameters of the cleaning procedure; they should be 27 | % passed in as Name-Value Pairs. If the method removes too many (or too few) channels, time 28 | % windows, or general high-amplitude ("burst") artifacts, you will want to tune these values. 29 | % Hopefully you only need to do this in rare cases. 30 | % 31 | % ChannelCriterion : Minimum channel correlation. If a channel is correlated at less than this 32 | % value to an estimate based on other channels, it is considered abnormal in 33 | % the given time window. This method requires that channel locations are 34 | % available and roughly correct; otherwise a fallback criterion will be used. 35 | % (default: 0.85) 36 | % 37 | % LineNoiseCriterion : If a channel has more line noise relative to its signal than this value, in 38 | % standard deviations based on the total channel population, it is considered 39 | % abnormal. (default: 4) 40 | % 41 | % BurstCriterion : Standard deviation cutoff for removal of bursts (via ASR). Data portions whose 42 | % variance is larger than this threshold relative to the calibration data are 43 | % considered missing data and will be removed. The most aggressive value that can 44 | % be used without losing much EEG is 3. For new users it is recommended to at 45 | % first visually inspect the difference between the original and cleaned data to 46 | % get a sense of the removed content at various levels. An agressive value is 5 47 | % and a quite conservative value is 20. Default: 5 (from the GUI, default is 20). 48 | % 49 | % WindowCriterion : Criterion for removing time windows that were not repaired completely. This may 50 | % happen if the artifact in a window was composed of too many simultaneous 51 | % uncorrelated sources (for example, extreme movements such as jumps). This is 52 | % the maximum fraction of contaminated channels that are tolerated in the final 53 | % output data for each considered window. Generally a lower value makes the 54 | % criterion more aggressive. Default: 0.25. Reasonable range: 0.05 (very 55 | % aggressive) to 0.3 (very lax). 56 | % 57 | % Highpass : Transition band for the initial high-pass filter in Hz. This is formatted as 58 | % [transition-start, transition-end]. Default: [0.25 0.75]. 59 | % 60 | % NOTE: The following are detail parameters that may be tuned if one of the criteria does 61 | % not seem to be doing the right thing. These basically amount to side assumptions about the 62 | % data that usually do not change much across recordings, but sometimes do. 63 | % 64 | % ChannelCriterionMaxBadTime : This is the maximum tolerated fraction of the recording duration 65 | % during which a channel may be flagged as "bad" without being 66 | % removed altogether. Generally a lower (shorter) value makes the 67 | % criterion more aggresive. Reasonable range: 0.15 (very aggressive) 68 | % to 0.6 (very lax). Default: 0.5. 69 | % 70 | % BurstCriterionRefMaxBadChns: If a number is passed in here, the ASR method will be calibrated based 71 | % on sufficiently clean data that is extracted first from the 72 | % recording that is then processed with ASR. This number is the 73 | % maximum tolerated fraction of "bad" channels within a given time 74 | % window of the recording that is considered acceptable for use as 75 | % calibration data. Any data windows within the tolerance range are 76 | % then used for calibrating the threshold statistics. Instead of a 77 | % number one may also directly pass in a data set that contains 78 | % calibration data (for example a minute of resting EEG). 79 | % 80 | % If this is set to 'off', all data is used for calibration. This will 81 | % work as long as the fraction of contaminated data is lower than the 82 | % the breakdown point of the robust statistics in the ASR 83 | % calibration (50%, where 30% of clearly recognizable artifacts is a 84 | % better estimate of the practical breakdown point). 85 | % 86 | % A lower value makes this criterion more aggressive. Reasonable 87 | % range: 0.05 (very aggressive) to 0.3 (quite lax). If you have lots 88 | % of little glitches in a few channels that don't get entirely 89 | % cleaned you might want to reduce this number so that they don't go 90 | % into the calibration data. Default: 0.075. 91 | % 92 | % BurstCriterionRefTolerances : These are the power tolerances outside of which a channel in a 93 | % given time window is considered "bad", in standard deviations 94 | % relative to a robust EEG power distribution (lower and upper 95 | % bound). Together with the previous parameter this determines how 96 | % ASR calibration data is be extracted from a recording. Can also be 97 | % specified as 'off' to achieve the same effect as in the previous 98 | % parameter. Default: [-Inf 5.5]. 99 | % 100 | % BurstRejection : 'on' or 'off'. If 'on' reject portions of data containing burst instead of 101 | % correcting them using ASR. Default is 'off'. 102 | % 103 | % WindowCriterionTolerances : These are the power tolerances outside of which a channel in the final 104 | % output data is considered "bad", in standard deviations relative 105 | % to a robust EEG power distribution (lower and upper bound). Any time 106 | % window in the final (repaired) output which has more than the 107 | % tolerated fraction (set by the WindowCriterion parameter) of channel 108 | % with a power outside of this range will be considered incompletely 109 | % repaired and will be removed from the output. This last stage can be 110 | % skipped either by setting the WindowCriterion to 'off' or by taking 111 | % the third output of this processing function (which does not include 112 | % the last stage). Default: [-Inf 7]. 113 | % 114 | % FlatlineCriterion : Maximum tolerated flatline duration. In seconds. If a channel has a longer 115 | % flatline than this, it will be considered abnormal. Default: 5 116 | % 117 | % NoLocsChannelCriterion : Criterion for removing bad channels when no channel locations are 118 | % present. This is a minimum correlation value that a given channel must 119 | % have w.r.t. a fraction of other channels. A higher value makes the 120 | % criterion more aggressive. Reasonable range: 0.4 (very lax) - 0.6 121 | % (quite aggressive). Default: 0.45. 122 | % 123 | % NoLocsChannelCriterionExcluded : The fraction of channels that must be sufficiently correlated with 124 | % a given channel for it to be considered "good" in a given time 125 | % window. Applies only to the NoLocsChannelCriterion. This adds 126 | % robustness against pairs of channels that are shorted or other 127 | % that are disconnected but record the same noise process. 128 | % Reasonable range: 0.1 (fairly lax) to 0.3 (very aggressive); 129 | % note that increasing this value requires the ChannelCriterion 130 | % to be relaxed in order to maintain the same overall amount of 131 | % removed channels. Default: 0.1. 132 | % 133 | % MaxMem : The maximum amount of memory in MB used by the algorithm when processing. 134 | % See function asr_process for more information. Default is 64. 135 | % 136 | % Out: 137 | % EEG : Final cleaned EEG recording. 138 | % 139 | % HP : Optionally just the high-pass filtered data. 140 | % 141 | % BUR : Optionally the data without final removal of "irrecoverable" windows. 142 | % 143 | % Examples: 144 | % % Load a recording, clean it, and visualize the difference (using the defaults) 145 | % raw = pop_loadset(...); 146 | % clean = clean_artifacts(raw); 147 | % vis_artifacts(clean,raw); 148 | % 149 | % % Use a more aggressive threshold (passing the parameters in by position) 150 | % raw = pop_loadset(...); 151 | % clean = clean_artifacts(raw,[],2.5); 152 | % vis_artifacts(clean,raw); 153 | % 154 | % % Passing some parameter by name (here making the WindowCriterion setting less picky) 155 | % raw = pop_loadset(...); 156 | % clean = clean_artifacts(raw,'WindowCriterion',0.25); 157 | % 158 | % % Disabling the WindowCriterion and ChannelCriterion altogether 159 | % raw = pop_loadset(...); 160 | % clean = clean_artifacts(raw,'WindowCriterion','off','ChannelCriterion','off'); 161 | % 162 | % 163 | % Christian Kothe, Swartz Center for Computational Neuroscience, UCSD 164 | % 2012-09-04 165 | 166 | % Copyright (C) Christian Kothe, SCCN, 2012, ckothe@ucsd.edu 167 | % 168 | % This program is free software; you can redistribute it and/or modify it under the terms of the GNU 169 | % General Public License as published by the Free Software Foundation; either version 2 of the 170 | % License, or (at your option) any later version. 171 | % 172 | % This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without 173 | % even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 174 | % General Public License for more details. 175 | % 176 | % You should have received a copy of the GNU General Public License along with this program; if not, 177 | % write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 178 | % USA 179 | 180 | hlp_varargin2struct(varargin,... 181 | {'chancorr_crit','ChannelCorrelationCriterion','ChannelCriterion'}, 0.8, ... 182 | {'line_crit','LineNoiseCriterion'}, 4, ... 183 | {'burst_crit','BurstCriterion'}, 5, ... 184 | {'fusechanrej','Fusechanrej'}, [], ... % unused in this function 185 | {'channels','Channels'}, [], ... 186 | {'channels_ignore','Channels_ignore'}, [], ... 187 | {'window_crit','WindowCriterion'}, 0.25, ... 188 | {'num_samples','NumSamples'}, 50, ... 189 | {'highpass_band','Highpass'}, [0.25 0.75], ... 190 | {'channel_crit_maxbad_time','ChannelCriterionMaxBadTime'}, 0.5, ... 191 | {'burst_crit_refmaxbadchns','BurstCriterionRefMaxBadChns'}, 0.075, ... 192 | {'burst_crit_reftolerances','BurstCriterionRefTolerances'}, [-inf 5.5], ... 193 | {'distance2','Distance'}, 'euclidian', ... 194 | {'window_crit_tolerances','WindowCriterionTolerances'},[-inf 7], ... 195 | {'burst_rejection','BurstRejection'},'off', ... 196 | {'nolocs_channel_crit','NoLocsChannelCriterion'}, 0.45, ... 197 | {'nolocs_channel_crit_excluded','NoLocsChannelCriterionExcluded'}, 0.1, ... 198 | {'max_mem','MaxMem'}, 64, ... 199 | {'availableRAM_GB','availableRAM_GB'}, NaN, ... 200 | {'flatline_crit','FlatlineCriterion'}, 5); 201 | 202 | if ~isnan(availableRAM_GB) 203 | max_mem = availableRAM_GB*1000; 204 | end 205 | 206 | % ignore some channels 207 | if ~isempty(channels) 208 | if ~isempty(channels_ignore) 209 | error('Can include or ignore channel but not both at the same time') 210 | end 211 | if ~iscell(channels) 212 | error('Cannot exclude channels without channel labels') 213 | end 214 | oriEEG = EEG; 215 | EEG = pop_select(EEG, 'channel', channels); 216 | oriEEG_without_ignored_channels = EEG; 217 | EEG.event = []; % will be added back later 218 | end 219 | if ~isempty(channels_ignore) 220 | if ~iscell(channels_ignore) 221 | error('Cannot exclude channels without channel labels') 222 | end 223 | oriEEG = EEG; 224 | EEG = pop_select(EEG, 'nochannel', channels_ignore); 225 | oriEEG_without_ignored_channels = EEG; 226 | EEG.event = []; % will be added back later 227 | end 228 | 229 | % remove flat-line channels 230 | if ~strcmp(flatline_crit,'off') 231 | disp('Detecting flat line...') 232 | EEG = clean_flatlines(EEG,flatline_crit); 233 | end 234 | 235 | % high-pass filter the data 236 | if ~strcmp(highpass_band,'off') 237 | disp('Applying highpass filter...') 238 | EEG = clean_drifts(EEG,highpass_band); 239 | end 240 | if nargout > 1 241 | HP = EEG; 242 | end 243 | 244 | % remove noisy channels by correlation and line-noise thresholds 245 | oldSrate = EEG.srate; 246 | EEG.srate = round(EEG.srate); 247 | removed_channels = []; 248 | if ~strcmp(chancorr_crit,'off') || ~strcmp(line_crit,'off') %#ok 249 | if strcmp(chancorr_crit,'off') 250 | chancorr_crit = 0; end 251 | if strcmp(line_crit,'off') 252 | line_crit = 100; end 253 | try 254 | [EEG,removed_channels] = clean_channels(EEG,chancorr_crit,line_crit,[],channel_crit_maxbad_time, num_samples); 255 | catch e 256 | % if strcmp(e.identifier,'clean_channels:bad_chanlocs') 257 | disp('Your dataset appears to lack correct channel locations; using a location-free channel cleaning method.'); 258 | [EEG,removed_channels] = clean_channels_nolocs(EEG,nolocs_channel_crit,nolocs_channel_crit_excluded,[],channel_crit_maxbad_time); 259 | % else 260 | % rethrow(e); 261 | % end 262 | end 263 | end 264 | 265 | % repair bursts by ASR 266 | if ~strcmp(burst_crit,'off') 267 | if ~strcmpi(distance2, 'euclidian') 268 | BUR = clean_asr(EEG,burst_crit,[],[],[],burst_crit_refmaxbadchns,burst_crit_reftolerances,[], [], true, max_mem); 269 | else 270 | try 271 | BUR = clean_asr(EEG,burst_crit,[],[],[],burst_crit_refmaxbadchns,burst_crit_reftolerances,[], [], false, max_mem); 272 | catch 273 | lasterr 274 | return 275 | end 276 | end 277 | 278 | if strcmp(burst_rejection,'on') 279 | % portion of data which have changed 280 | sample_mask = sum(abs(EEG.data-BUR.data),1) < 1e-8; 281 | 282 | % find latency of regions 283 | retain_data_intervals = reshape(find(diff([false sample_mask false])),2,[])'; 284 | retain_data_intervals(:,2) = retain_data_intervals(:,2)-1; 285 | 286 | % remove small intervals 287 | if ~isempty(retain_data_intervals) 288 | smallIntervals = diff(retain_data_intervals')' < 5; 289 | for iInterval = find(smallIntervals)' 290 | sample_mask(retain_data_intervals(iInterval,1):retain_data_intervals(iInterval,2)) = 0; 291 | end 292 | retain_data_intervals(smallIntervals,:) = []; 293 | end 294 | 295 | % reject regions 296 | EEG = pop_select(EEG, 'point', retain_data_intervals); 297 | EEG.etc.clean_sample_mask = sample_mask; 298 | 299 | % check 300 | try 301 | res = checksamples(EEG); 302 | if res.CleanBoundaryMatch == 0 303 | fprintf(2, 'Your data is fine, but upon double checking the removed portions of data, there is\n'); 304 | fprintf(2, 'a small discrepency; if you can reproduce the warning message, send us your dataset.\n'); 305 | end 306 | catch, end 307 | else 308 | EEG = BUR; 309 | end 310 | end 311 | EEG.srate = oldSrate; 312 | 313 | if nargout > 2 314 | BUR = EEG; 315 | end 316 | 317 | % remove irrecoverable time windows based on power 318 | if ~strcmp(window_crit,'off') && ~strcmp(window_crit_tolerances,'off') 319 | disp('Now doing final post-cleanup of the output.'); 320 | EEG = clean_windows(EEG,window_crit,window_crit_tolerances); 321 | end 322 | disp('Use vis_artifacts to compare the cleaned data to the original.'); 323 | 324 | % add back original channels 325 | if ~isempty(channels) || ~isempty(channels_ignore) 326 | if ~isempty(removed_channels) 327 | removed_channels = { oriEEG_without_ignored_channels.chanlocs(removed_channels).labels }; 328 | end 329 | 330 | % Apply same transformation to the data before removal of channels and data 331 | EEG = eeg_checkset(EEG, 'eventconsistency'); 332 | if ~isempty(EEG.event) && isfield(EEG.event, 'type') && isstr(EEG.event(1).type) 333 | disp('Adding back removed channels'); 334 | 335 | boundaryEvents = eeg_findboundaries(EEG); 336 | 337 | % remove again data portions 338 | if ~isempty(boundaryEvents) 339 | boundloc = [ EEG.event(boundaryEvents).latency ]; 340 | dur = [ EEG.event(boundaryEvents).duration ]; 341 | cumdur = cumsum(dur); 342 | boundloc = boundloc + [0 cumdur(1:end-1) ]; 343 | boundloc = [ boundloc; boundloc+dur-1]'; 344 | oriEEG = eeg_eegrej(oriEEG, ceil(boundloc)); 345 | end 346 | 347 | % copy clean data to oriEEG (in case data was corrected 348 | [~,chanInds1, chanInds2] = intersect({ oriEEG.chanlocs.labels }, { EEG.chanlocs.labels }); 349 | if size(oriEEG.data,2) ~= size(EEG.data,2) 350 | error('Issue with adding back removed channels. Remove channels, then remove bad portions of data.'); 351 | end 352 | oriEEG.data(chanInds1,:) = EEG.data(chanInds2,:); 353 | oriEEG.pnts = EEG.pnts; 354 | 355 | EEG = oriEEG; 356 | if ~isempty(removed_channels) 357 | EEG = pop_select(EEG, 'rmchannel', removed_channels); 358 | end 359 | end 360 | 361 | end 362 | 363 | %creates a structure summarizing if Assumptions 2 and 4 are true or false 364 | %for each dataset in my study 365 | function CleanSampleMismatch = check_rm_samples(EEG) 366 | 367 | CleanSampleMismatch.oldLength = length(EEG.etc.clean_sample_mask); 368 | CleanSampleMismatch.newLength = EEG.pnts; 369 | CleanSampleMismatch.maskSum = sum(EEG.etc.clean_sample_mask); 370 | 371 | if EEG.pnts ~= sum(EEG.etc.clean_sample_mask) %tests Assumption 2 372 | CleanSampleMismatch.CleanMaskMatch = 0; 373 | else 374 | CleanSampleMismatch.CleanMaskMatch = 1; 375 | end 376 | 377 | CleanSampleMismatch.boundaryDuration = 0; 378 | for j = 1: length(EEG.event) %finds total duration of boundary events 379 | if strcmp(EEG.event(j).type, 'boundary') 380 | CleanSampleMismatch.boundaryDuration = CleanSampleMismatch.boundaryDuration + EEG.event(j).duration; 381 | end 382 | end 383 | CleanSampleMismatch.nonBoundaryLength = length(EEG.etc.clean_sample_mask) - CleanSampleMismatch.boundaryDuration; 384 | 385 | if EEG.pnts ~= CleanSampleMismatch.nonBoundaryLength %tests Assumption 4 386 | CleanSampleMismatch.CleanBoundaryMatch = 0; 387 | else 388 | CleanSampleMismatch.CleanBoundaryMatch = 1; 389 | end 390 | 391 | -------------------------------------------------------------------------------- /clean_asr.m: -------------------------------------------------------------------------------- 1 | function signal = clean_asr(signal,cutoff,windowlen,stepsize,maxdims,ref_maxbadchannels,ref_tolerances,ref_wndlen,usegpu,useriemannian,maxmem) 2 | % Run the ASR method on some high-pass filtered recording. 3 | % Signal = clean_asr(Signal,StandardDevCutoff,WindowLength,BlockSize,MaxDimensions,ReferenceMaxBadChannels,RefTolerances,ReferenceWindowLength,UseGPU,UseRiemannian,MaxMem) 4 | % 5 | % This is an automated artifact rejection function that ensures that the data contains no events 6 | % that have abnormally strong power; the subspaces on which those events occur are reconstructed 7 | % (interpolated) based on the rest of the EEG signal during these time periods. 8 | % 9 | % The basic principle is to first find a section of data that represents clean "reference" EEG and 10 | % to compute statistics on there. Then, the function goes over the whole data in a sliding window 11 | % and finds the subspaces in which there is activity that is more than a few standard deviations 12 | % away from the reference EEG (this threshold is a tunable parameter). Once the function has found 13 | % the bad subspaces it will treat them as missing data and reconstruct their content using a mixing 14 | % matrix that was calculated on the clean data. 15 | % 16 | % Notes: 17 | % This function by default attempts to use the Statistics toolbox in order to automatically 18 | % extract calibration data for use by ASR from the given recording. This step is automatically 19 | % skipped if no Statistics toolbox is present (then the entire recording will be used for 20 | % calibration, which is fine for mildly contaminated data -- see ReferenceMaxBadChannels below). 21 | % 22 | % In: 23 | % Signal : continuous data set, assumed to be *zero mean*, e.g., appropriately high-passed (e.g. 24 | % >0.5Hz or with a 0.5Hz - 1.0Hz transition band) 25 | % 26 | % Cutoff : Standard deviation cutoff for removal of bursts (via ASR). Data portions whose variance 27 | % is larger than this threshold relative to the calibration data are considered missing 28 | % data and will be removed. The most aggressive value that can be used without losing 29 | % much EEG is 3. For new users it is recommended to at first visually inspect the difference 30 | % between the original and cleaned data to get a sense of the removed content at various 31 | % levels. An aggressive value is 5, and conservative value is 20. Default: 5. 32 | % 33 | % The following are detail parameters that usually do not have to be tuned. If you cannot get 34 | % the function to do what you want, you might consider adapting these better to your data. 35 | % 36 | % WindowLength : Length of the statistcs window, in seconds. This should not be much longer 37 | % than the time scale over which artifacts persist, but the number of samples in 38 | % the window should not be smaller than 1.5x the number of channels. Default: 39 | % max(0.5,1.5*Signal.nbchan/Signal.srate); 40 | % 41 | % StepSize : Step size for processing. The reprojection matrix will be updated every this many 42 | % samples and a blended matrix is used for the in-between samples. If empty this will 43 | % be set the WindowLength/2 in samples. Default: [] 44 | % 45 | % MaxDimensions : Maximum dimensionality to reconstruct. Up to this many dimensions (or up to this 46 | % fraction of dimensions) can be reconstructed for a given data segment. This is 47 | % since the lower eigenvalues are usually not estimated very well. Default: 2/3. 48 | % 49 | % ReferenceMaxBadChannels : If a number is passed in here, the ASR method will be calibrated based 50 | % on sufficiently clean data that is extracted first from the recording 51 | % that is then processed with ASR. This number is the maximum tolerated 52 | % fraction of "bad" channels within a given time window of the recording 53 | % that is considered acceptable for use as calibration data. Any data 54 | % windows within the tolerance range are then used for calibrating the 55 | % threshold statistics. Instead of a number one may also directly pass 56 | % in a data set that contains calibration data (for example a minute of 57 | % resting EEG) or the name of a data set in the workspace. 58 | % 59 | % If this is set to 'off', all data is used for calibration. This will 60 | % work as long as the fraction of contaminated data is lower than the 61 | % the breakdown point of the robust statistics in the ASR calibration 62 | % (50%, where 30% of clearly recognizable artifacts is a better estimate 63 | % of the practical breakdown point). 64 | % 65 | % A lower value makes this criterion more aggressive. Reasonable range: 66 | % 0.05 (very aggressive) to 0.3 (quite lax). If you have lots of little 67 | % glitches in a few channels that don't get entirely cleaned you might 68 | % want to reduce this number so that they don't go into the calibration 69 | % data. Default: 0.075. 70 | % 71 | % 72 | % ReferenceTolerances : These are the power tolerances outside of which a channel in a 73 | % given time window is considered "bad", in standard deviations relative to 74 | % a robust EEG power distribution (lower and upper bound). Together with the 75 | % previous parameter this determines how ASR calibration data is be 76 | % extracted from a recording. Can also be specified as 'off' to achieve the 77 | % same effect as in the previous parameter. Default: [-3.5 5.5]. 78 | % 79 | % ReferenceWindowLength : Granularity at which EEG time windows are extracted 80 | % for calibration purposes, in seconds. Default: 1. 81 | % 82 | % UseRiemannian : [true|false] Use Riemannian distance instead of Euclidian distance. 83 | % Riemannian distance used the modication in the following publication 84 | % Blum Sarah, Jacobsen Nadine S. J., Bleichner Martin G., Debener Stefan (2019) 85 | % A Riemannian Modification of Artifact Subspace Reconstruction for EEG Artifact 86 | % Handling, Frontiers in Human Neuroscience, 13, 141. DOI=10.3389/fnhum.2019.00141. 87 | % 88 | % MaxMem : Amount of memory to use. See asr_process for more information. 89 | % 90 | % UseGPU : Whether to run on the GPU. This makes sense for offline processing if you have a a card with 91 | % enough memory and good double-precision performance (e.g., NVIDIA GTX Titan or K20). 92 | % Note that for this to work you need to a) have the Parallel Computing toolbox and b) remove 93 | % the dummy gather.m file from the path. Default: false 94 | % 95 | % Out: 96 | % Signal : data set with local peaks removed 97 | % 98 | % Examples: 99 | % % use the defaults 100 | % eeg = clean_asr(eeg); 101 | % 102 | % % use a more aggressive threshold 103 | % eeg = clean_asr(eeg,2.5); 104 | % 105 | % % disable subset selection of calibration data (use all data instead) 106 | % eeg = clean_asr(eeg,[],[],[],[],'off'); 107 | % 108 | % % use a custom calibration measurement (e.g., EEGLAB dataset containing a baseline recording) 109 | % eeg = clean_asr(eeg,[],[],[],[],mybaseline); 110 | % 111 | % Christian Kothe, Swartz Center for Computational Neuroscience, UCSD 112 | % 2012-10-15 113 | 114 | % Copyright (C) Christian Kothe, SCCN, 2012, ckothe@ucsd.edu 115 | % 116 | % This program is free software; you can redistribute it and/or modify it under the terms of the GNU 117 | % General Public License as published by the Free Software Foundation; either version 2 of the 118 | % License, or (at your option) any later version. 119 | % 120 | % This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without 121 | % even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 122 | % General Public License for more details. 123 | % 124 | % You should have received a copy of the GNU General Public License along with this program; if not, 125 | % write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 126 | % USA 127 | 128 | if ~exist('cutoff','var') || isempty(cutoff) cutoff = 5; end 129 | if ~exist('windowlen','var') || isempty(windowlen) windowlen = max(0.5,1.5*signal.nbchan/signal.srate); end 130 | if ~exist('stepsize','var') || isempty(stepsize) stepsize = []; end 131 | if ~exist('maxdims','var') || isempty(maxdims) maxdims = 0.66; end 132 | if ~exist('ref_maxbadchannels','var') || isempty(ref_maxbadchannels) ref_maxbadchannels = 0.075; end 133 | if ~exist('ref_tolerances','var') || isempty(ref_tolerances) ref_tolerances = [-3.5 5.5]; end 134 | if ~exist('ref_wndlen','var') || isempty(ref_wndlen) ref_wndlen = 1; end 135 | if ~exist('usegpu','var') || isempty(usegpu) usegpu = false; end 136 | if ~exist('maxmem','var') || isempty(maxmem) maxmem = 64; end 137 | if ~exist('useriemannian','var') || isempty(useriemannian) useriemannian = false; end 138 | 139 | signal.data = double(signal.data); 140 | 141 | % first determine the reference (calibration) data 142 | if isnumeric(ref_maxbadchannels) && isnumeric(ref_tolerances) && isnumeric(ref_wndlen) 143 | disp('Finding a clean section of the data...'); 144 | try 145 | ref_section = clean_windows(signal,ref_maxbadchannels,ref_tolerances,ref_wndlen); 146 | catch e 147 | disp('An error occurred while trying to identify a subset of clean calibration data from the recording.'); 148 | disp('If this is because do not have EEGLAB loaded or no Statistics toolbox, you can generally'); 149 | disp('skip this step by passing in ''off'' as the ReferenceMaxBadChannels parameter.'); 150 | disp('Error details: '); 151 | hlp_handleerror(e,1); 152 | disp('Falling back to using the entire data for calibration.') 153 | ref_section = signal; 154 | end 155 | elseif strcmp(ref_maxbadchannels,'off') || strcmp(ref_tolerances,'off') || strcmp(ref_wndlen,'off') 156 | disp('Using the entire data for calibration (reference parameters set to ''off'').') 157 | ref_section = signal; 158 | elseif ischar(ref_maxbadchannels) && isvarname(ref_maxbadchannels) 159 | disp('Using a user-supplied data set in the workspace.'); 160 | ref_section = evalin('base',ref_maxbadchannels); 161 | elseif all(isfield(ref_maxbadchannels,{'data','srate','chanlocs'})) 162 | disp('Using a user-supplied clean section of data.'); 163 | ref_section = ref_maxbadchannels; 164 | else 165 | error('Unsupported value for argument ref_maxbadchannels.'); 166 | end 167 | 168 | % calibrate on the reference data 169 | disp('Estimating calibration statistics; this may take a while...'); 170 | if exist('hlp_diskcache','file') 171 | if useriemannian 172 | state = hlp_diskcache('filterdesign',@asr_calibrate_r,ref_section.data,ref_section.srate,cutoff); 173 | else 174 | state = hlp_diskcache('filterdesign',@asr_calibrate,ref_section.data,ref_section.srate,cutoff); 175 | end 176 | else 177 | if useriemannian 178 | state = asr_calibrate_r(ref_section.data,ref_section.srate,cutoff, [], [], [], [], [], [], [], maxmem); 179 | else 180 | state = asr_calibrate(ref_section.data,ref_section.srate,cutoff, [], [], [], [], [], [], [], maxmem); 181 | end 182 | end 183 | clear ref_section; 184 | 185 | if isempty(stepsize) 186 | stepsize = floor(signal.srate*windowlen/2); end 187 | 188 | % extrapolate last few samples of the signal 189 | sig = [signal.data bsxfun(@minus,2*signal.data(:,end),signal.data(:,(end-1):-1:end-round(windowlen/2*signal.srate)))]; 190 | % process signal using ASR 191 | if useriemannian 192 | [signal.data,state] = asr_process_r(sig,signal.srate,state,windowlen,windowlen/2,stepsize,maxdims,maxmem,usegpu); 193 | else 194 | [signal.data,state] = asr_process(sig,signal.srate,state,windowlen,windowlen/2,stepsize,maxdims,maxmem,usegpu); 195 | end 196 | % shift signal content back (to compensate for processing delay) 197 | signal.data(:,1:size(state.carry,2)) = []; 198 | -------------------------------------------------------------------------------- /clean_channels.m: -------------------------------------------------------------------------------- 1 | function [signal,removed_channels] = clean_channels(signal,corr_threshold,noise_threshold,window_len,max_broken_time,num_samples,subset_size) 2 | % Remove channels with abnormal data from a continuous data set. 3 | % Signal = clean_channels(Signal,CorrelationThreshold,LineNoiseThreshold,WindowLength,MaxBrokenTime,NumSamples,SubsetSize) 4 | % 5 | % This is an automated artifact rejection function which ensures that the data contains no channels 6 | % that record only noise for extended periods of time. If channels with control signals are 7 | % contained in the data these are usually also removed. The criterion is based on correlation: if a 8 | % channel has lower correlation to its robust estimate (based on other channels) than a given threshold 9 | % for a minimum period of time (or percentage of the recording), it will be removed. 10 | % 11 | % In: 12 | % Signal : Continuous data set, assumed to be appropriately high-passed (e.g. >0.5Hz or 13 | % with a 0.5Hz - 2.0Hz transition band). 14 | % 15 | % CorrelationThreshold : Correlation threshold. If a channel is correlated at less than this value 16 | % to its robust estimate (based on other channels), it is considered abnormal in 17 | % the given time window. Default: 0.85. 18 | % 19 | % LineNoiseThreshold : If a channel has more line noise relative to its signal than this value, in 20 | % standard deviations from the channel population mean, it is considered abnormal. 21 | % Default: 4. 22 | % 23 | % 24 | % The following are detail parameters that usually do not have to be tuned. If you cannot get 25 | % the function to do what you want, you might consider adapting these to your data. 26 | % 27 | % WindowLength : Length of the windows (in seconds) for which correlation is computed; ideally 28 | % short enough to reasonably capture periods of global artifacts or intermittent 29 | % sensor dropouts, but not shorter (for statistical reasons). Default: 5. 30 | % 31 | % MaxBrokenTime : Maximum time (either in seconds or as fraction of the recording) during which a 32 | % retained channel may be broken. Reasonable range: 0.1 (very aggressive) to 0.6 33 | % (very lax). The default is 0.4. 34 | % 35 | % NumSamples : Number of RANSAC samples. This is the number of samples to generate in the random 36 | % sampling consensus process. The larger this value, the more robust but also slower 37 | % the processing will be. Default: 50. 38 | % 39 | % SubsetSize : Subset size. This is the size of the channel subsets to use for robust reconstruction, 40 | % as a fraction of the total number of channels. Default: 0.25. 41 | % 42 | % Out: 43 | % Signal : data set with bad channels removed 44 | % 45 | % Christian Kothe, Swartz Center for Computational Neuroscience, UCSD 46 | % 2014-05-12 47 | 48 | % Copyright (C) Christian Kothe, SCCN, 2014, christian@sccn.ucsd.edu 49 | % 50 | % This program is free software; you can redistribute it and/or modify it under the terms of the GNU 51 | % General Public License as published by the Free Software Foundation; either version 2 of the 52 | % License, or (at your option) any later version. 53 | % 54 | % This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without 55 | % even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 56 | % General Public License for more details. 57 | % 58 | % You should have received a copy of the GNU General Public License along with this program; if not, 59 | % write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 60 | % USA 61 | 62 | if ~exist('corr_threshold','var') || isempty(corr_threshold) corr_threshold = 0.8; end 63 | if ~exist('noise_threshold','var') || isempty(noise_threshold) noise_threshold = 4; end 64 | if ~exist('window_len','var') || isempty(window_len) window_len = 5; end 65 | if ~exist('max_broken_time','var') || isempty(max_broken_time) max_broken_time = 0.4; end 66 | if ~exist('num_samples','var') || isempty(num_samples) num_samples = 50; end 67 | if ~exist('subset_size','var') || isempty(subset_size) subset_size = 0.25; end 68 | if ~exist('reset_rng','var') || isempty(reset_rng) reset_rng = true; end 69 | 70 | subset_size = round(subset_size*size(signal.data,1)); 71 | 72 | % flag channels 73 | if max_broken_time > 0 && max_broken_time < 1 %#ok<*NODEF> 74 | max_broken_time = size(signal.data,2)*max_broken_time; 75 | else 76 | max_broken_time = round(signal.srate)*max_broken_time; 77 | end 78 | 79 | signal.data = double(signal.data); 80 | [C,S] = size(signal.data); 81 | window_len = window_len*round(signal.srate); 82 | wnd = 0:window_len-1; 83 | offsets = 1:window_len:S-window_len; 84 | W = length(offsets); 85 | 86 | fprintf('Scanning for bad channels...\n'); 87 | 88 | if signal.srate > 100 89 | % remove signal content above 50Hz 90 | B = design_fir(100,[2*[0 45 50]/signal.srate 1],[1 1 0 0]); 91 | for c=signal.nbchan:-1:1 92 | X(:,c) = filtfilt_fast(B,1,signal.data(c,:)'); end 93 | % determine z-scored level of EM noise-to-signal ratio for each channel 94 | noisiness = mad(signal.data'-X)./mad(X,1); 95 | znoise = (noisiness - median(noisiness)) ./ (mad(noisiness,1)*1.4826); 96 | % trim channels based on that 97 | noise_mask = znoise > noise_threshold; 98 | else 99 | X = signal.data'; 100 | noise_mask = false(C,1)'; % transpose added. Otherwise gives an error below at removed_channels = removed_channels | noise_mask'; (by Ozgur Balkan) 101 | end 102 | 103 | if ~(isfield(signal.chanlocs,'X') && isfield(signal.chanlocs,'Y') && isfield(signal.chanlocs,'Z') && all([length([signal.chanlocs.X]),length([signal.chanlocs.Y]),length([signal.chanlocs.Z])] > length(signal.chanlocs)*0.5)) 104 | error('clean_channels:bad_chanlocs','To use this function most of your channels should have X,Y,Z location measurements.'); end 105 | 106 | % get the matrix of all channel locations [3xN] 107 | [x,y,z] = deal({signal.chanlocs.X},{signal.chanlocs.Y},{signal.chanlocs.Z}); 108 | usable_channels = find(~cellfun('isempty',x) & ~cellfun('isempty',y) & ~cellfun('isempty',z)); 109 | locs = [cell2mat(x(usable_channels));cell2mat(y(usable_channels));cell2mat(z(usable_channels))]; 110 | X = X(:,usable_channels); 111 | 112 | % caculate all-channel reconstruction matrices from random channel subsets 113 | if reset_rng 114 | rng('default') 115 | end 116 | if exist('OCTAVE_VERSION', 'builtin') == 0 117 | P = hlp_microcache('cleanchans',@calc_projector,locs,num_samples,subset_size); 118 | else 119 | P = calc_projector(locs,num_samples,subset_size); 120 | end 121 | corrs = zeros(length(usable_channels),W); 122 | 123 | % calculate each channel's correlation to its RANSAC reconstruction for each window 124 | timePassedList = zeros(W,1); 125 | for o=1:W 126 | tic; % makoto 127 | XX = X(offsets(o)+wnd,:); 128 | YY = sort(reshape(XX*P,length(wnd),length(usable_channels),num_samples),3); 129 | YY = YY(:,:,round(end/2)); 130 | corrs(:,o) = sum(XX.*YY)./(sqrt(sum(XX.^2)).*sqrt(sum(YY.^2))); 131 | timePassedList(o) = toc; % makoto 132 | medianTimePassed = median(timePassedList(1:o)); 133 | fprintf('clean_channel: %3.0d/%d blocks, %.1f minutes remaining.\n', o, W, medianTimePassed*(W-o)/60); % makoto 134 | end 135 | 136 | flagged = corrs < corr_threshold; 137 | 138 | % mark all channels for removal which have more flagged samples than the maximum number of 139 | % ignored samples 140 | removed_channels = false(C,1); 141 | removed_channels(usable_channels) = sum(flagged,2)*window_len > max_broken_time; 142 | removed_channels = removed_channels | noise_mask'; 143 | 144 | % apply removal 145 | if mean(removed_channels) > 0.75 146 | error('clean_channels:bad_chanlocs','More than 75%% of your channels were removed -- this is probably caused by incorrect channel location measurements (e.g., wrong cap design).'); 147 | elseif any(removed_channels) 148 | try 149 | signal = pop_select(signal,'nochannel',find(removed_channels)); 150 | catch e 151 | if ~exist('pop_select','file') 152 | disp('Apparently you do not have EEGLAB''s pop_select() on the path.'); 153 | else 154 | disp('Could not select channels using EEGLAB''s pop_select(); details: '); 155 | hlp_handleerror(e,1); 156 | end 157 | fprintf('Removing %i channels and dropping signal meta-data.\n',nnz(removed_channels)); 158 | if length(signal.chanlocs) == size(signal.data,1) 159 | signal.chanlocs = signal.chanlocs(~removed_channels); end 160 | signal.data = signal.data(~removed_channels,:); 161 | signal.nbchan = size(signal.data,1); 162 | [signal.icawinv,signal.icasphere,signal.icaweights,signal.icaact,signal.stats,signal.specdata,signal.specicaact] = deal([]); 163 | end 164 | if isfield(signal.etc,'clean_channel_mask') 165 | signal.etc.clean_channel_mask(signal.etc.clean_channel_mask) = ~removed_channels; 166 | else 167 | signal.etc.clean_channel_mask = ~removed_channels; 168 | end 169 | end 170 | 171 | 172 | 173 | % calculate a bag of reconstruction matrices from random channel subsets 174 | function P = calc_projector(locs,num_samples,subset_size) 175 | %stream = RandStream('mt19937ar','Seed',435656); 176 | rand_samples = {}; 177 | for k=num_samples:-1:1 178 | tmp = zeros(size(locs,2)); 179 | subset = randsample(1:size(locs,2),subset_size); 180 | % subset = randsample(1:size(locs,2),subset_size,stream); 181 | tmp(subset,:) = real(sphericalSplineInterpolate(locs(:,subset),locs))'; 182 | rand_samples{k} = tmp; 183 | end 184 | P = horzcat(rand_samples{:}); 185 | 186 | 187 | function Y = randsample(X,num) 188 | Y = []; 189 | while length(Y) 204 | Y = median(abs(bsxfun(@minus,X,median(X)))); 205 | -------------------------------------------------------------------------------- /clean_channels_nolocs.m: -------------------------------------------------------------------------------- 1 | function [signal,removed_channels] = clean_channels_nolocs(signal,min_corr,ignored_quantile,window_len,max_broken_time,linenoise_aware) 2 | % Remove channels with abnormal data from a continuous data set. 3 | % Signal = clean_channels_nolocs(Signal,MinCorrelation,IgnoredQuantile,WindowLength,MaxBrokenTime,LineNoiseAware) 4 | % 5 | % This is an automated artifact rejection function which ensures that the data contains no channels 6 | % that record only noise for extended periods of time. If channels with control signals are 7 | % contained in the data these are usually also removed. The criterion is based on correlation: if a 8 | % channel is decorrelated from all others (pairwise correlation < a given threshold), excluding a 9 | % given fraction of most correlated channels -- and if this holds on for a sufficiently long fraction 10 | % of the data set -- then the channel is removed. 11 | % 12 | % In: 13 | % Signal : Continuous data set, assumed to be appropriately high-passed (e.g. >0.5Hz or 14 | % with a 0.5Hz - 2.0Hz transition band). 15 | % 16 | % MinCorrelation : Minimum correlation between a channel and any other channel (in a short period 17 | % of time) below which the channel is considered abnormal for that time period. 18 | % Reasonable range: 0.4 (very lax) to 0.6 (quite aggressive). The default is 0.45. 19 | % 20 | % 21 | % The following are detail parameters that usually do not have to be tuned. If you cannot get 22 | % the function to do what you want, you might consider adapting these to your data. 23 | % 24 | % IgnoredQuantile : Fraction of channels that need to have at least the given MinCorrelation value 25 | % w.r.t. the channel under consideration. This allows to deal with channels or 26 | % small groups of channels that measure the same noise source, e.g. if they are 27 | % shorted. If many channels can be disconnected during an experiment and you 28 | % have strong noise in the room, you might increase this fraction, but consider 29 | % that this a) requires you to decrease the MinCorrelation appropriately and b) 30 | % this can make the correlation measure more brittle. Reasonable range: 0.05 (rather 31 | % lax) to 0.2 (very tolerant re disconnected/shorted channels).The default is 32 | % 0.1. 33 | % 34 | % WindowLength : Length of the windows (in seconds) for which correlation is computed; ideally 35 | % short enough to reasonably capture periods of global artifacts (which are 36 | % ignored), but not shorter (for statistical reasons). Default: 2. 37 | % 38 | % MaxBrokenTime : Maximum time (either in seconds or as fraction of the recording) during which a 39 | % retained channel may be broken. Reasonable range: 0.1 (very aggressive) to 0.6 40 | % (very lax). The default is 0.5. 41 | % 42 | % LineNoiseAware : Whether the operation should be performed in a line-noise aware manner. If enabled, 43 | % the correlation measure will not be affected by the presence or absence of line 44 | % noise (using a temporary notch filter). Default: true. 45 | % 46 | % Out: 47 | % Signal : data set with bad channels removed 48 | % 49 | % Notes: 50 | % This function requires the Signal Processing toolbox. 51 | % 52 | % Christian Kothe, Swartz Center for Computational Neuroscience, UCSD 53 | % 2010-07-06 54 | 55 | % Copyright (C) Christian Kothe, SCCN, 2010, christian@sccn.ucsd.edu 56 | % 57 | % This program is free software; you can redistribute it and/or modify it under the terms of the GNU 58 | % General Public License as published by the Free Software Foundation; either version 2 of the 59 | % License, or (at your option) any later version. 60 | % 61 | % This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without 62 | % even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 63 | % General Public License for more details. 64 | % 65 | % You should have received a copy of the GNU General Public License along with this program; if not, 66 | % write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 67 | % USA 68 | 69 | if ~exist('min_corr','var') || isempty(min_corr) min_corr = 0.45; end 70 | if ~exist('ignored_quantile','var') || isempty(ignored_quantile) ignored_quantile = 0.1; end 71 | if ~exist('window_len','var') || isempty(window_len) window_len = 2; end 72 | if ~exist('max_broken_time','var') || isempty(max_broken_time) max_broken_time = 0.5; end 73 | if ~exist('linenoise_aware','var') || isempty(linenoise_aware) linenoise_aware = true; end 74 | 75 | 76 | % flag channels 77 | if max_broken_time > 0 && max_broken_time < 1 %#ok<*NODEF> 78 | max_broken_time = size(signal.data,2)*max_broken_time; 79 | else 80 | max_broken_time = signal.srate*max_broken_time; 81 | end 82 | 83 | signal.data = double(signal.data); 84 | [C,S] = size(signal.data); 85 | window_len = window_len*signal.srate; 86 | wnd = 0:window_len-1; 87 | offsets = 1:window_len:S-window_len; 88 | W = length(offsets); 89 | retained = 1:(C-ceil(C*ignored_quantile)); 90 | 91 | % optionally ignore both 50 and 60 Hz spectral components... 92 | if linenoise_aware 93 | Bwnd = design_kaiser(2*45/signal.srate,2*50/signal.srate,60,true); 94 | if signal.srate <= 110 95 | error('Sampling rate must be above 110 Hz'); 96 | elseif signal.srate <= 130 97 | B = design_fir(length(Bwnd)-1,[2*[0 45 50 55]/signal.srate 1],[1 1 0 1 1],[],Bwnd); 98 | else 99 | B = design_fir(length(Bwnd)-1,[2*[0 45 50 55 60 65]/signal.srate 1],[1 1 0 1 0 1 1],[],Bwnd); 100 | end 101 | for c=signal.nbchan:-1:1 102 | X(:,c) = filtfilt_fast(B,1,signal.data(c,:)'); end 103 | else 104 | X = signal.data'; 105 | end 106 | 107 | % for each window, flag channels with too low correlation to any other channel (outside the 108 | % ignored quantile) 109 | flagged = zeros(C,W); 110 | for o=1:W 111 | sortcc = sort(abs(corrcoef(X(offsets(o)+wnd,:)))); 112 | flagged(:,o) = all(sortcc(retained,:) < min_corr); 113 | end 114 | 115 | % mark all channels for removal which have more flagged samples than the maximum number of 116 | % ignored samples 117 | removed_channels = sum(flagged,2)*window_len > max_broken_time; 118 | 119 | % apply removal 120 | if all(removed_channels) 121 | disp('Warning: all channels are flagged bad according to the used criterion: not removing anything.'); 122 | elseif any(removed_channels) 123 | disp('Now removing bad channels...'); 124 | try 125 | signal = pop_select(signal,'nochannel',find(removed_channels)); 126 | catch e 127 | if ~exist('pop_select','file') 128 | disp('Apparently you do not have EEGLAB''s pop_select() on the path.'); 129 | else 130 | disp('Could not select channels using EEGLAB''s pop_select(); details: '); 131 | hlp_handleerror(e,1); 132 | end 133 | disp('Falling back to a basic substitute and dropping signal meta-data.'); 134 | if length(signal.chanlocs) == size(signal.data,1) 135 | signal.chanlocs = signal.chanlocs(~removed_channels); end 136 | signal.data = signal.data(~removed_channels,:); 137 | signal.nbchan = size(signal.data,1); 138 | [signal.icawinv,signal.icasphere,signal.icaweights,signal.icaact,signal.stats,signal.specdata,signal.specicaact] = deal([]); 139 | end 140 | if isfield(signal.etc,'clean_channel_mask') && sum(signal.etc.clean_channel_mask) == length(removed_channels) 141 | signal.etc.clean_channel_mask(signal.etc.clean_channel_mask) = ~removed_channels; 142 | else 143 | signal.etc.clean_channel_mask = ~removed_channels; 144 | end 145 | end 146 | -------------------------------------------------------------------------------- /clean_drifts.m: -------------------------------------------------------------------------------- 1 | function signal = clean_drifts(signal,transition,attenuation) 2 | % Removes drifts from the data using a forward-backward high-pass filter. 3 | % Signal = clean_drifts(Signal,Transition) 4 | % 5 | % This removes drifts from the data using a forward-backward (non-causal) filter. 6 | % NOTE: If you are doing directed information flow analysis, do no use this filter but some other one. 7 | % 8 | % In: 9 | % Signal : the continuous data to filter 10 | % 11 | % Transition : the transition band in Hz, i.e. lower and upper edge of the transition 12 | % (default: [0.5 1]) 13 | % 14 | % Attenuation : stop-band attenuation, in db (default: 80) 15 | % 16 | % Out: 17 | % Signal : the filtered signal 18 | % 19 | % Notes: 20 | % This function requires the Signal Processing toolbox. 21 | % 22 | % Christian Kothe, Swartz Center for Computational Neuroscience, UCSD 23 | % 2012-09-01 24 | 25 | % Copyright (C) Christian Kothe, SCCN, 2012, ckothe@ucsd.edu 26 | % 27 | % This program is free software; you can redistribute it and/or modify it under the terms of the GNU 28 | % General Public License as published by the Free Software Foundation; either version 2 of the 29 | % License, or (at your option) any later version. 30 | % 31 | % This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without 32 | % even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 33 | % General Public License for more details. 34 | % 35 | % You should have received a copy of the GNU General Public License along with this program; if not, 36 | % write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 37 | % USA 38 | 39 | if ~exist('transition','var') || isempty(transition) transition = [0.5 1]; end 40 | if ~exist('attenuation','var') || isempty(attenuation) attenuation = 80; end 41 | signal.data = double(signal.data); 42 | 43 | % design highpass FIR filter 44 | transition = 2*transition/signal.srate; 45 | wnd = design_kaiser(transition(1),transition(2),attenuation,true); 46 | B = design_fir(length(wnd)-1,[0 transition 1],[0 0 1 1],[],wnd); 47 | 48 | % apply it, channel by channel to save memory 49 | signal.data = signal.data(:,:)'; 50 | for c=1:signal.nbchan 51 | signal.data(:,c) = filtfilt_fast(B,1,signal.data(:,c)); end 52 | signal.data = signal.data'; 53 | signal.etc.clean_drifts_kernel = B; 54 | -------------------------------------------------------------------------------- /clean_flatlines.m: -------------------------------------------------------------------------------- 1 | function signal = clean_flatlines(signal,max_flatline_duration,max_allowed_jitter) 2 | % Remove (near-) flat-lined channels. 3 | % Signal = clean_flatlines(Signal,MaxFlatlineDuration,MaxAllowedJitter) 4 | % 5 | % This is an automated artifact rejection function which ensures that 6 | % the data contains no flat-lined channels. 7 | % 8 | % In: 9 | % Signal : continuous data set, assumed to be appropriately high-passed (e.g. >0.5Hz or 10 | % with a 0.5Hz - 2.0Hz transition band) 11 | % 12 | % MaxFlatlineDuration : Maximum tolerated flatline duration. In seconds. If a channel has a longer 13 | % flatline than this, it will be considered abnormal. Default: 5 14 | % 15 | % MaxAllowedJitter : Maximum tolerated jitter during flatlines. As a multiple of epsilon. 16 | % Default: 20 17 | % 18 | % Out: 19 | % Signal : data set with flat channels removed 20 | % 21 | % Examples: 22 | % % use with defaults 23 | % eeg = clean_flatlines(eeg); 24 | % 25 | % Christian Kothe, Swartz Center for Computational Neuroscience, UCSD 26 | % 2012-08-30 27 | 28 | % Copyright (C) Christian Kothe, SCCN, 2012, ckothe@ucsd.edu 29 | % 30 | % This program is free software; you can redistribute it and/or modify it under the terms of the GNU 31 | % General Public License as published by the Free Software Foundation; either version 2 of the 32 | % License, or (at your option) any later version. 33 | % 34 | % This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without 35 | % even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 36 | % General Public License for more details. 37 | % 38 | % You should have received a copy of the GNU General Public License along with this program; if not, 39 | % write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 40 | % USA 41 | 42 | if ~exist('max_flatline_duration','var') || isempty(max_flatline_duration) max_flatline_duration = 5; end 43 | if ~exist('max_allowed_jitter','var') || isempty(max_allowed_jitter) max_allowed_jitter = 20; end 44 | 45 | % flag channels 46 | removed_channels = false(1,signal.nbchan); 47 | for c=1:signal.nbchan 48 | zero_intervals = reshape(find(diff([false abs(diff(signal.data(c,:)))<(max_allowed_jitter*eps) false])),2,[])'; 49 | if max(zero_intervals(:,2) - zero_intervals(:,1)) > max_flatline_duration*signal.srate 50 | removed_channels(c) = true; end 51 | end 52 | 53 | % remove them 54 | if all(removed_channels) 55 | disp('Warning: all channels have a flat-line portion; not removing anything.'); 56 | elseif any(removed_channels) 57 | disp('Now removing flat-line channels...'); 58 | try 59 | signal = pop_select(signal,'nochannel',find(removed_channels)); 60 | catch e 61 | if ~exist('pop_select','file') 62 | disp('Apparently you do not have EEGLAB''s pop_select() on the path.'); 63 | else 64 | disp('Could not select channels using EEGLAB''s pop_select(); details: '); 65 | hlp_handleerror(e,1); 66 | end 67 | disp('Falling back to a basic substitute and dropping signal meta-data.'); 68 | if length(signal.chanlocs) == size(signal.data,1) 69 | signal.chanlocs = signal.chanlocs(~removed_channels); end 70 | signal.data = signal.data(~removed_channels,:); 71 | signal.nbchan = size(signal.data,1); 72 | [signal.icawinv,signal.icasphere,signal.icaweights,signal.icaact,signal.stats,signal.specdata,signal.specicaact] = deal([]); 73 | end 74 | if isfield(signal.etc,'clean_channel_mask') 75 | signal.etc.clean_channel_mask(signal.etc.clean_channel_mask) = ~removed_channels; 76 | else 77 | signal.etc.clean_channel_mask = ~removed_channels; 78 | end 79 | end 80 | -------------------------------------------------------------------------------- /clean_rawdata.m: -------------------------------------------------------------------------------- 1 | % clean_rawdata(): a wrapper for EEGLAB to call clean_artifacts function 2 | % 3 | % Usage: 4 | % >> EEG = clean_rawdata(EEG, arg_flatline, arg_highpass, arg_channel, arg_noisy, arg_burst, arg_window) 5 | % 6 | % ------------------ below is from clean_artifacts ----------------------- 7 | % 8 | % This function removes flatline channels, low-frequency drifts, noisy channels, short-time bursts 9 | % and incompletely repaird segments from the data. Tip: Any of the core parameters can also be 10 | % passed in as [] to use the respective default of the underlying functions, or as 'off' to disable 11 | % it entirely. 12 | % 13 | % Hopefully parameter tuning should be the exception when using this function -- however, there are 14 | % 3 parameters governing how aggressively bad channels, bursts, and irrecoverable time windows are 15 | % being removed, plus several detail parameters that only need tuning under special circumstances. 16 | % 17 | % FlatlineCriterion: Maximum tolerated flatline duration. In seconds. If a channel has a longer 18 | % flatline than this, it will be considered abnormal. Default: 5 19 | % 20 | % Highpass : Transition band for the initial high-pass filter in Hz. This is formatted as 21 | % [transition-start, transition-end]. Default: [0.25 0.75]. 22 | % 23 | % ChannelCriterion : Minimum channel correlation. If a channel is correlated at less than this 24 | % value to a reconstruction of it based on other channels, it is considered 25 | % abnormal in the given time window. This method requires that channel 26 | % locations are available and roughly correct; otherwise a fallback criterion 27 | % will be used. (default: 0.85) 28 | % 29 | % LineNoiseCriterion : If a channel has more line noise relative to its signal than this value, in 30 | % standard deviations based on the total channel population, it is considered 31 | % abnormal. (default: 4) 32 | % 33 | % BurstCriterion : Standard deviation cutoff for removal of bursts (via ASR). Data portions whose 34 | % variance is larger than this threshold relative to the calibration data are 35 | % considered missing data and will be removed. The most aggressive value that can 36 | % be used without losing much EEG is 3. For new users it is recommended to at 37 | % first visually inspect the difference between the original and cleaned data to 38 | % get a sense of the removed content at various levels. An agressive value is 5 39 | % and a quite conservative value is 20. Default: 5 (from the GUI, default is 20). 40 | % 41 | % WindowCriterion : Criterion for removing time windows that were not repaired completely. This may 42 | % happen if the artifact in a window was composed of too many simultaneous 43 | % uncorrelated sources (for example, extreme movements such as jumps). This is 44 | % the maximum fraction of contaminated channels that are tolerated in the final 45 | % output data for each considered window. Generally a lower value makes the 46 | % criterion more aggressive. Default: 0.25. Reasonable range: 0.05 (very 47 | % aggressive) to 0.3 (very lax). 48 | % 49 | % see also: clean_artifacts 50 | 51 | % Author: Makoto Miyakoshi and Christian Kothe, SCCN,INC,UCSD 52 | % History: 53 | % 05/13/2014 ver 1.2 by Christian. Added better channel removal function (uses locations if available). 54 | % 07/16/2013 ver 1.1 by Makoto and Christian. Minor update for help and default values. 55 | % 06/26/2013 ver 1.0 by Makoto. Created. 56 | 57 | % Copyright (C) 2013, Makoto Miyakoshi and Christian Kothe, SCCN,INC,UCSD 58 | % 59 | % This program is free software; you can redistribute it and/or modify 60 | % it under the terms of the GNU General Public License as published by 61 | % the Free Software Foundation; either version 2 of the License, or 62 | % (at your option) any later version. 63 | % 64 | % This program is distributed in the hope that it will be useful, 65 | % but WITHOUT ANY WARRANTY; without even the implied warranty of 66 | % MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 67 | % GNU General Public License for more details. 68 | % 69 | % You should have received a copy of the GNU General Public License 70 | % along with this program; if not, write to the Free Software 71 | % Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA 72 | 73 | function cleanEEG = clean_rawdata(EEG, arg_flatline, arg_highpass, arg_channel, arg_noisy, arg_burst, arg_window) 74 | 75 | disp('The function clean_rawdata has been deprecated and is only kept for backward'); 76 | disp('compatibility. Use the clean_artifacts function instead.'); 77 | 78 | if arg_flatline == -1; arg_flatline = 'off'; disp('flatchan rej disabled.' ); end 79 | if arg_highpass == -1; arg_highpass = 'off'; disp('highpass disabled.' ); end 80 | if arg_channel == -1; arg_channel = 'off'; disp('badchan rej disabled.' ); end 81 | if arg_noisy == -1; arg_noisy = 'off'; disp('noise-based rej disabled.'); end 82 | if arg_burst == -1; arg_burst = 'off'; disp('burst clean disabled.' ); end 83 | if arg_window == -1; arg_window = 'off'; disp('bad window rej disabled.'); end 84 | 85 | cleanEEG = clean_artifacts(EEG, 'FlatlineCriterion', arg_flatline,... 86 | 'Highpass', arg_highpass,... 87 | 'ChannelCriterion', arg_channel,... 88 | 'LineNoiseCriterion', arg_noisy,... 89 | 'BurstCriterion', arg_burst,... 90 | 'WindowCriterion', arg_window); 91 | -------------------------------------------------------------------------------- /clean_windows.m: -------------------------------------------------------------------------------- 1 | function [signal,sample_mask] = clean_windows(signal,max_bad_channels,zthresholds,window_len,window_overlap,max_dropout_fraction,min_clean_fraction,truncate_quant,step_sizes,shape_range) 2 | % Remove periods with abnormally high-power content from continuous data. 3 | % [Signal,Mask] = clean_windows(Signal,MaxBadChannels,PowerTolerances,WindowLength,WindowOverlap,MaxDropoutFraction,Min) 4 | % 5 | % This function cuts segments from the data which contain high-power artifacts. Specifically, 6 | % only windows are retained which have less than a certain fraction of "bad" channels, where a channel 7 | % is bad in a window if its power is above or below a given upper/lower threshold (in standard 8 | % deviations from a robust estimate of the EEG power distribution in the channel). 9 | % 10 | % In: 11 | % Signal : Continuous data set, assumed to be appropriately high-passed (e.g. >1Hz or 12 | % 0.5Hz - 2.0Hz transition band) 13 | % 14 | % MaxBadChannels : The maximum number or fraction of bad channels that a retained window may still 15 | % contain (more than this and it is removed). Reasonable range is 0.05 (very clean 16 | % output) to 0.3 (very lax cleaning of only coarse artifacts). Default: 0.2. 17 | % 18 | % PowerTolerances: The minimum and maximum standard deviations within which the RMS of a channel 19 | % must lie (relative to a robust estimate of the clean EEG RMS distribution in 20 | % the channel) for it to be considered "not bad". Default: [-3.5 5]. 21 | % 22 | % The following are detail parameters that usually do not have to be tuned. If you can't get 23 | % the function to do what you want, you might consider adapting these to your data. 24 | % 25 | % WindowLength : Window length that is used to check the data for artifact content. This is 26 | % ideally as long as the expected time scale of the artifacts but not shorter 27 | % than half a cycle of the high-pass filter that was used. Default: 1. 28 | % 29 | % WindowOverlap : Window overlap fraction. The fraction of two successive windows that overlaps. 30 | % Higher overlap ensures that fewer artifact portions are going to be missed (but 31 | % is slower). (default: 0.66) 32 | % 33 | % MaxDropoutFraction : Maximum fraction that can have dropouts. This is the maximum fraction of 34 | % time windows that may have arbitrarily low amplitude (e.g., due to the 35 | % sensors being unplugged). (default: 0.1) 36 | % 37 | % MinCleanFraction : Minimum fraction that needs to be clean. This is the minimum fraction of time 38 | % windows that need to contain essentially uncontaminated EEG. (default: 0.25) 39 | % 40 | % 41 | % The following are expert-level parameters that you should not tune unless you fully understand 42 | % how the method works. 43 | % 44 | % TruncateQuantile : Truncated Gaussian quantile. Quantile range [upper,lower] of the truncated 45 | % Gaussian distribution that shall be fit to the EEG contents. (default: [0.022 0.6]) 46 | % 47 | % StepSizes : Grid search stepping. Step size of the grid search, in quantiles; separately for 48 | % [lower,upper] edge of the truncated Gaussian. The lower edge has finer stepping 49 | % because the clean data density is assumed to be lower there, so small changes in 50 | % quantile amount to large changes in data space. (default: [0.01 0.01]) 51 | % 52 | % ShapeRange : Shape parameter range. Search range for the shape parameter of the generalized 53 | % Gaussian distribution used to fit clean EEG. (default: 1.7:0.15:3.5) 54 | % 55 | % Out: 56 | % Signal : data set with bad time periods removed. 57 | % 58 | % Mask : mask of retained samples (logical array) 59 | % 60 | % Christian Kothe, Swartz Center for Computational Neuroscience, UCSD 61 | % 2010-07-06 62 | 63 | % Copyright (C) Christian Kothe, SCCN, 2010, ckothe@ucsd.edu 64 | % 65 | % History 66 | % 04/26/2017 Makoto. Changed action When EEG.etc.clean_sample_mask is present, 67 | % 68 | % This program is free software; you can redistribute it and/or modify it under the terms of the GNU 69 | % General Public License as published by the Free Software Foundation; either version 2 of the 70 | % License, or (at your option) any later version. 71 | % 72 | % This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without 73 | % even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 74 | % General Public License for more details. 75 | % 76 | % You should have received a copy of the GNU General Public License along with this program; if not, 77 | % write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 78 | % USA 79 | 80 | % handle inputs 81 | if ~exist('max_bad_channels','var') || isempty(max_bad_channels) max_bad_channels = 0.2; end 82 | if ~exist('zthresholds','var') || isempty(zthresholds) zthresholds = [-3.5 5]; end 83 | if ~exist('window_len','var') || isempty(window_len) window_len = 1; end 84 | if ~exist('window_overlap','var') || isempty(window_overlap) window_overlap = 0.66; end 85 | if ~exist('max_dropout_fraction','var') || isempty(max_dropout_fraction) max_dropout_fraction = 0.1; end 86 | if ~exist('min_clean_fraction','var') || isempty(min_clean_fraction) min_clean_fraction = 0.25; end 87 | if ~exist('truncate_quant','var') || isempty(truncate_quant) truncate_quant = [0.022 0.6]; end 88 | if ~exist('step_sizes','var') || isempty(step_sizes) step_sizes = [0.01 0.01]; end 89 | if ~exist('shape_range','var') || isempty(shape_range) shape_range = 1.7:0.15:3.5; end 90 | if ~isempty(max_bad_channels) && max_bad_channels > 0 && max_bad_channels < 1 %#ok<*NODEF> 91 | max_bad_channels = round(size(signal.data,1)*max_bad_channels); end 92 | 93 | signal.data = double(signal.data); 94 | [C,S] = size(signal.data); 95 | N = window_len*signal.srate; 96 | wnd = 0:N-1; 97 | offsets = round(1:N*(1-window_overlap):S-N); 98 | 99 | fprintf('Determining time window rejection thresholds...'); 100 | % for each channel... 101 | for c = C:-1:1 102 | % compute RMS amplitude for each window... 103 | X = signal.data(c,:).^2; 104 | X = sqrt(sum(X(bsxfun(@plus,offsets,wnd')))/N); 105 | % robustly fit a distribution to the clean EEG part 106 | [mu,sig] = fit_eeg_distribution(X, ... 107 | min_clean_fraction, max_dropout_fraction, ... 108 | truncate_quant, step_sizes,shape_range); 109 | % calculate z scores relative to that 110 | wz(c,:) = (X - mu)/sig; 111 | end 112 | disp('done.'); 113 | 114 | % sort z scores into quantiles 115 | swz = sort(wz); 116 | % determine which windows to remove 117 | remove_mask = false(1,size(swz,2)); 118 | if max(zthresholds)>0 119 | remove_mask(swz(end-max_bad_channels,:) > max(zthresholds)) = true; end 120 | if min(zthresholds)<0 121 | remove_mask(swz(1+max_bad_channels,:) < min(zthresholds)) = true; end 122 | removed_windows = find(remove_mask); 123 | 124 | % find indices of samples to remove 125 | removed_samples = repmat(offsets(removed_windows)',1,length(wnd))+repmat(wnd,length(removed_windows),1); 126 | % mask them out 127 | sample_mask = true(1,S); 128 | sample_mask(removed_samples(:)) = false; 129 | fprintf('Keeping %.1f%% (%.0f seconds) of the data.\n',100*(mean(sample_mask)),nnz(sample_mask)/signal.srate); 130 | % determine intervals to retain 131 | retain_data_intervals = reshape(find(diff([false sample_mask false])),2,[])'; 132 | retain_data_intervals(:,2) = retain_data_intervals(:,2)-1; 133 | 134 | % apply selection 135 | try 136 | signal = pop_select(signal, 'point', retain_data_intervals); 137 | catch e 138 | if ~exist('pop_select','file') 139 | disp('Apparently you do not have EEGLAB''s pop_select() on the path.'); 140 | else 141 | disp('Could not select time windows using EEGLAB''s pop_select(); details: '); 142 | hlp_handleerror(e,1); 143 | end 144 | %disp('Falling back to a basic substitute and dropping signal meta-data.'); 145 | warning('Falling back to a basic substitute and dropping signal meta-data.'); 146 | signal.data = signal.data(:,sample_mask); 147 | signal.pnts = size(signal.data,2); 148 | signal.xmax = signal.xmin + (signal.pnts-1)/signal.srate; 149 | [signal.event,signal.urevent,signal.epoch,signal.icaact,signal.reject,signal.stats,signal.specdata,signal.specicaact] = deal(signal.event([]),signal.urevent([]),[],[],[],[],[],[]); 150 | end 151 | % if isfield(signal.etc,'clean_sample_mask') 152 | % signal.etc.clean_sample_mask(signal.etc.clean_sample_mask) = sample_mask; 153 | % else 154 | % signal.etc.clean_sample_mask = sample_mask; 155 | % end 156 | if isfield(signal.etc,'clean_sample_mask') 157 | oneInds = find(signal.etc.clean_sample_mask == 1); 158 | if length(oneInds) == length(sample_mask) 159 | signal.etc.clean_sample_mask(oneInds) = sample_mask; 160 | else 161 | disp('Warning: EEG.etc.clean_sample is present. It is overwritten.'); 162 | end 163 | else 164 | signal.etc.clean_sample_mask = sample_mask; 165 | end 166 | -------------------------------------------------------------------------------- /eegplugin_clean_rawdata.m: -------------------------------------------------------------------------------- 1 | % eegplugin_clean_rawdata() - a wrapper to plug-in Christian's clean_artifact() into EEGLAB. . 2 | % 3 | % Usage: 4 | % >> eegplugin_firstPassOutlierTrimmer(fig,try_strings,catch_strings); 5 | % 6 | % see also: clean_artifacts 7 | 8 | % Author: Makoto Miyakoshi and Christian Kothe, SCCN,INC,UCSD 2013 9 | % History: 10 | % 05/13/2014 ver 0.30 by Christian. Added better channel removal function that uses channel locations if present. 11 | % 11/20/2013 ver 0.20 by Christian. Updated signal processing routines to current versions. 12 | % 11/15/2013 ver 0.12 by Christian. Fixed a rare bug in asr_process. 13 | % 07/16/2013 ver 0.11 by Makoto. Minor change on the name on GUI menu. 14 | % 06/26/2013 ver 0.10 by Makoto. Created. 15 | 16 | % This program is free software; you can redistribute it and/or modify 17 | % it under the terms of the GNU General Public License as published by 18 | % the Free Software Foundation; either version 2 of the License, or 19 | % (at your option) any later version. 20 | % 21 | % This program is distributed in the hope that it will be useful, 22 | % but WITHOUT ANY WARRANTY; without even the implied warranty of 23 | % MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 24 | % GNU General Public License for more details. 25 | % 26 | % You should have received a copy of the GNU General Public License 27 | % along with this program; if not, write to the Free Software 28 | % Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA 29 | 30 | function vers = eegplugin_clean_rawdata(fig,try_strings,catch_strings) 31 | 32 | vers = '2.10'; 33 | cmd = [ try_strings.check_data ... 34 | '[EEG,LASTCOM] = pop_clean_rawdata(EEG);' ... 35 | catch_strings.new_and_hist ]; 36 | 37 | p = fileparts(which('eegplugin_clean_rawdata')); 38 | addpath(fullfile(p, 'manopt')); 39 | addpath(fullfile(p, 'manopt', 'manopt','manifolds', 'grassmann')); 40 | addpath(fullfile(p, 'manopt', 'manopt','tools')); 41 | addpath(fullfile(p, 'manopt', 'manopt','core')); 42 | addpath(fullfile(p, 'manopt', 'manopt','solvers', 'trustregions')); 43 | 44 | % create menu 45 | toolsmenu = findobj(fig, 'tag', 'tools'); 46 | uimenu( toolsmenu, 'label', 'Reject data using Clean Rawdata and ASR', 'userdata', 'startup:off;epoch:off;study:on', ... 47 | 'callback', cmd, 'position', 7); 48 | -------------------------------------------------------------------------------- /gui_interface.ai: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sccn/clean_rawdata/1cb4d154adf102ac99cc07b7335d564aada9ee57/gui_interface.ai -------------------------------------------------------------------------------- /gui_interface.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sccn/clean_rawdata/1cb4d154adf102ac99cc07b7335d564aada9ee57/gui_interface.png -------------------------------------------------------------------------------- /pop_clean_rawdata.m: -------------------------------------------------------------------------------- 1 | % pop_clean_rawdata(): Launches GUI to collect user inputs for clean_artifacts(). 2 | % ASR stands for artifact subspace reconstruction. 3 | % To disable method(s), enter -1. 4 | % Usage: 5 | % >> EEG = pop_clean_rawdata(EEG); 6 | % 7 | % see also: clean_artifacts 8 | 9 | % Author: Arnaud Delorme, Makoto Miyakoshi and Christian Kothe, SCCN,INC,UCSD 10 | % History: 11 | % 07/2019. Reprogrammed from Scratch (Arnaud Delorme) 12 | % 07/31/2018 Makoto. Returns error if input data size is 3. 13 | % 04/26/2017 Makoto. Deletes existing EEG.etc.clean_channel/sample_mask. Try-catch to skip potential error in vis_artifact. 14 | % 07/18/2014 ver 1.4 by Makoto and Christian. New channel removal method supported. str2num -> str2num due to str2num([a b]) == NaN. 15 | % 11/08/2013 ver 1.3 by Makoto. Menu words changed. asr_process() line 168 bug fixed. 16 | % 10/07/2013 ver 1.2 by Makoto. Help implemented. History bug fixed. 17 | % 07/16/2013 ver 1.1 by Makoto and Christian. Minor update for help and default values. 18 | % 06/26/2013 ver 1.0 by Makoto. Created. 19 | 20 | % Copyright (C) 2013, Arnaud Delorme, Makoto Miyakoshi and Christian Kothe, SCCN,INC,UCSD 21 | % 22 | % This program is free software; you can redistribute it and/or modify 23 | % it under the terms of the GNU General Public License as published by 24 | % the Free Software Foundation; either version 2 of the License, or 25 | % (at your option) any later version. 26 | % 27 | % This program is distributed in the hope that it will be useful, 28 | % but WITHOUT ANY WARRANTY; without even the implied warranty of 29 | % MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 30 | % GNU General Public License for more details. 31 | % 32 | % You should have received a copy of the GNU General Public License 33 | % along with this program; if not, write to the Free Software 34 | % Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA 35 | 36 | function [EEG, com] = pop_clean_rawdata(EEG, varargin) 37 | 38 | % Check input 39 | com = ''; 40 | if ndims(EEG(1).data) == 3 41 | error('Input data must be continuous. This data seems epoched.') 42 | end 43 | 44 | fusechanrej = false; 45 | if nargin < 2 46 | % Obtain user inputs. 47 | cb_filter = 'if get(gcbo, ''value''), set(findobj(gcbf, ''userdata'', ''filter''), ''enable'', ''on''); else set(findobj(gcbf, ''userdata'', ''filter''), ''enable'', ''off'', ''value'', 0); end'; 48 | cb_chan = 'if get(gcbo, ''value''), set(findobj(gcbf, ''userdata'', ''chan'') , ''enable'', ''on''); else set(findobj(gcbf, ''userdata'', ''chan'') , ''enable'', ''off'', ''value'', 0); end'; 49 | cb_asr = 'if get(gcbo, ''value''), set(findobj(gcbf, ''userdata'', ''asr'') , ''enable'', ''on''); else set(findobj(gcbf, ''userdata'', ''asr'') , ''enable'', ''off''); end'; 50 | cb_rej = 'if get(gcbo, ''value''), set(findobj(gcbf, ''userdata'', ''rej'') , ''enable'', ''on''); else set(findobj(gcbf, ''userdata'', ''rej'') , ''enable'', ''off''); end'; 51 | cb_select1 = 'pop_chansel(get(gcbf, ''userdata''), ''field'', ''labels'', ''handle'', findobj(''parent'', gcbf, ''tag'', ''chanuse'')); set(findobj(''parent'', gcbf, ''tag'', ''chanuseflag''), ''value'', 1); set(findobj(''parent'', gcbf, ''tag'', ''chanignoreflag''), ''value'', 0);'; 52 | cb_select2 = 'pop_chansel(get(gcbf, ''userdata''), ''field'', ''labels'', ''handle'', findobj(''parent'', gcbf, ''tag'', ''chanignore'')); set(findobj(''parent'', gcbf, ''tag'', ''chanignoreflag''), ''value'', 1); set(findobj(''parent'', gcbf, ''tag'', ''chanuseflag''), ''value'', 0);'; 53 | winsize = max(0.5,1.5*EEG(1).nbchan/EEG(1).srate); 54 | uilist = {... 55 | {'style' 'checkbox' 'string' 'Remove channel drift (data not already high-pass filtered)' 'fontweight' 'bold' 'tag' 'filter' 'callback' cb_filter} ... 56 | {} {'style' 'text' 'string' 'Linear filter (FIR) transition band [lo hi] in Hz ' 'userdata' 'filter' 'enable' 'off' } ... 57 | {'style' 'edit' 'string' '0.25 0.75', 'enable' 'off' 'tag','filterfreqs', 'userdata' 'filter' 'tooltipstring', wordwrap('The first number is the frequency below which everything is removed, and the second number is the frequency above which everything is retained. There is a linear transition in between. For best performance of subsequent processing steps the upper frequency should be close to 1 or 2 Hz, but you can go lower if certain activities need to be retained.',80)} ... 58 | ... 59 | {} ... 60 | {'style' 'checkbox' 'string' 'Process/remove channels' 'fontweight' 'bold' 'tag' 'chanrm' 'callback' cb_chan 'value' 1 } ... 61 | ... 62 | {} { 'style' 'checkbox' 'string' 'Only consider these channels' 'tag' 'chanuseflag' 'userdata' 'chan' } ... 63 | { 'style' 'pushbutton' 'string' '...' 'userdata' 'chan' 'callback' cb_select1 } ... 64 | {'style' 'edit' 'string' '' 'userdata' 'chan' 'tag' 'chanuse' } ... 65 | ... 66 | {} {'style' 'checkbox' 'string' 'Ignore these channels (ECG, EMG, ...)' 'userdata' 'chan' 'value' 0 'tag' 'chanignoreflag' } ... 67 | {'style' 'pushbutton' 'string' '...', 'userdata' 'chan' 'callback' cb_select2 } ... 68 | {'style' 'edit' 'string' '', 'userdata' 'chan' 'tag' 'chanignore' } ... 69 | ... 70 | {} {'style' 'checkbox' 'string' 'Remove channel if it is flat for more than (seconds)' 'tag' 'rmflat' 'userdata' 'chan' 'value' 1 } ... 71 | {'style' 'edit' 'string' '5', 'userdata' 'chan' 'tag' 'rmflatsec' 'tooltipstring', wordwrap('If a channel has a longer flatline than this, it will be removed. In seconds.',80)} ... 72 | ... 73 | {} {'style' 'checkbox' 'string' 'Max acceptable high-frequency noise std dev' 'value' 1 'tag' 'rmnoise' 'userdata' 'chan' } ... 74 | {'style' 'edit' 'string' '4', 'userdata' 'chan' 'tag' 'rmnoiseval' 'tooltipstring', wordwrap('If a channel has more line noise relative to its signal than this value, in standard deviations relative to the overall channel population, it will be removed.',80)} ... 75 | ... 76 | {} {'style' 'checkbox' 'string' 'Min acceptable correlation with nearby chans [0-1]' 'value' 1 'tag' 'rmcorr' 'userdata' 'chan' } ... 77 | {'style' 'edit' 'string' '0.8', 'userdata' 'chan' 'tag' 'rmcorrval' 'tooltipstring', wordwrap('If a channel has lower correlation than this to an estimate of its activity based on other channels, and this applies to more than half of the recording, the channel will be removed. This method requires that channel locations are available and roughly correct; otherwise a fallback criterion will tried used using a default setting; you can customize the fallback method by directly calling clean_channels_nolocs in the command line.',80)} ... 78 | ... 79 | {} {'style' 'checkbox' 'string' 'Perform Artifact Subspace Reconstruction bad burst correction/rejection' 'fontweight' 'bold' 'value' 1 'tag' 'asr' 'callback' cb_asr } ... 80 | {} {'style' 'text' 'string' sprintf('Max acceptable %1.1f second window std dev', winsize) 'value' 1 'userdata' 'asr' } ... 81 | {'style' 'edit' 'string' '20', 'tag' 'asrstdval' 'userdata' 'asr' 'tooltipstring', wordwrap('Standard deviation cutoff for removal of bursts. Data portions whose variance is larger than this threshold relative to the calibration data are considered missing data and will be removed. The most aggressive value that can be used without losing much EEG is 3. A reasonably conservative value is 5, but some extreme EEG bursts (e.g., sleep spindles) can cross even 5. For new users it is recommended to at first visually inspect the difference between the original and cleaned data to get a sense of the removed content at various levels.',80)} ... 82 | {} {'style' 'checkbox' 'string' 'Use Riemanian distance metric (not Euclidean) - beta' 'userdata' 'asr' 'value' 0 'tag' 'distance' } {} ... 83 | {} {'style' 'checkbox' 'tag' 'asrrej' 'string' 'Remove bad data periods (when uncheck, correct using ASR)' 'value' 1 'userdata' 'asr'} {} ... 84 | ... 85 | {} {'style' 'checkbox' 'string' 'Additional removal of bad data periods' 'fontweight' 'bold' 'value' 1 'tag' 'rejwin' 'callback' cb_rej } ... 86 | {} {'style' 'text' 'tag' 'asrwintext' 'string' 'Acceptable [min max] channel RMS range (+/- std dev)' 'userdata' 'rej'} ... 87 | {'style' 'edit' 'string' '-Inf 7','tag', 'rejwinval1', 'userdata' 'rej' 'tooltipstring', wordwrap('If a time window has a larger fraction of simultaneously corrupted channels than this (after the other cleaning attempts), it will be cut out of the data. This can happen if a time window was corrupted beyond the point where it could be recovered.',80)} ... 88 | {} {'style' 'text' 'tag' 'asrwintext' 'string' 'Maximum out-of-bound channels (%)' 'userdata' 'rej'} ... 89 | {'style' 'edit' 'string' '25','tag', 'rejwinval2', 'userdata' 'rej' 'tooltipstring', wordwrap('If a time window has a larger fraction of simultaneously corrupted channels than this (after the other cleaning attempts), it will be cut out of the data. This can happen if a time window was corrupted beyond the point where it could be recovered.',80)} ... 90 | ... 91 | {} {'style' 'checkbox' 'string' 'Pop up scrolling data window with rejected data highlighted' 'tag' 'vis' 'value' fastif(length(EEG) > 1, 0, 1) 'enable' fastif(length(EEG) > 1, 'off', 'on') }}; 92 | 93 | % channel labels 94 | % -------------- 95 | if ~isempty(EEG(1).chanlocs) 96 | tmpchanlocs = EEG(1).chanlocs; 97 | else 98 | tmpchanlocs = []; 99 | for index = 1:EEG(1).nbchan 100 | tmpchanlocs(index).labels = int2str(index); 101 | tmpchanlocs(index).type = ''; 102 | end 103 | end 104 | 105 | row = [0.1 1 0.3]; 106 | row4 = [0.1 0.8 0.2 0.3]; 107 | row2 = [0.1 1.2 0.1]; 108 | row3 = [0.9 0.2 0.3]; 109 | geom = { 1 row 1 1 row4 row4 row row row 1 1 row row2 row2 1 1 row row 1 1 }; 110 | geomvert = [ 1 1 0.3 1 1 1 1 1 1 0.3 1 1 1 1 0.3 1 1 1 0.3 1 ]; 111 | 112 | if length(EEG) > 1 113 | uilist = { uilist{1:6} {} {'style' 'checkbox' 'string' 'Fuse channel rejection for datasets with same subject and session' 'userdata' 'chan' 'fontweight' 'bold' 'tag' 'commonrej' 'value' 1} {} uilist{7:end} }; 114 | geom = [ geom(1:4) { row2 } geom(5:end)]; 115 | geomvert = [ geomvert(1:4) 1 geomvert(5:end) ]; 116 | end 117 | 118 | [res,~,~,outs] = inputgui('title', 'pop_clean_rawdata()', 'geomvert', geomvert, 'geometry', geom, 'uilist',uilist, 'helpcom', 'pophelp(''clean_artifacts'');', 'userdata', tmpchanlocs); 119 | 120 | % Return error if no input. 121 | if isempty(res) return; end 122 | 123 | % process multiple datasets 124 | % ------------------------- 125 | options = {}; 126 | opt.FlatlineCriterion = 'off'; 127 | opt.ChannelCriterion = 'off'; 128 | opt.LineNoiseCriterion = 'off'; 129 | opt.Highpass = 'off'; 130 | opt.BurstCriterion = 'off'; 131 | opt.WindowCriterion = 'off'; 132 | opt.BurstRejection = 'off'; 133 | opt.Distance = 'Euclidian'; 134 | 135 | if outs.filter, opt.Highpass = str2num(outs.filterfreqs); end 136 | 137 | if outs.chanrm 138 | if outs.chanignoreflag 139 | [ chaninds, chanlist ] = eeg_decodechan(EEG(1).chanlocs, outs.chanignore); 140 | if isempty(chanlist), chanlist = chaninds; end 141 | opt.channels_ignore = chanlist; 142 | end 143 | if outs.chanuseflag 144 | [ chaninds, chanlist ] = eeg_decodechan(EEG(1).chanlocs, outs.chanuse); 145 | if isempty(chanlist), chanlist = chaninds; end 146 | opt.channels = chanlist; 147 | end 148 | if outs.rmflat, opt.FlatlineCriterion = str2num(outs.rmflatsec); end 149 | if outs.rmcorr, opt.ChannelCriterion = str2num(outs.rmcorrval); end 150 | if outs.rmnoise, opt.LineNoiseCriterion = str2num(outs.rmnoiseval); end 151 | end 152 | 153 | if outs.asr 154 | opt.BurstCriterion = str2num(outs.asrstdval); 155 | if outs.distance, opt.Distance = 'Riemannian'; end 156 | end 157 | 158 | if outs.rejwin 159 | opt.WindowCriterionTolerances = str2num(outs.rejwinval1); 160 | opt.WindowCriterion = str2num(outs.rejwinval2)/100; 161 | end 162 | if outs.asrrej && ~strcmpi(opt.BurstCriterion, 'off') 163 | opt.BurstRejection = 'on'; 164 | end 165 | if isfield(outs,'commonrej') && outs.commonrej 166 | opt.fusechanrej = outs.commonrej; 167 | fusechanrej = true; 168 | end 169 | 170 | % convert structure to cell 171 | options = fieldnames(opt); 172 | options(:,2) = struct2cell(opt); 173 | options = options'; 174 | options = options(:)'; 175 | else 176 | options = varargin; 177 | for iOpt = 1:2:length(options) 178 | if strcmpi(options{iOpt}, 'fusechanrej') 179 | fusechanrej = true; 180 | end 181 | end 182 | end 183 | 184 | if length(EEG) > 1 185 | % process multiple datasets 186 | if nargin < 2 187 | [ EEG, com ] = eeg_eval( 'clean_artifacts', EEG, 'warning', 'on', 'params', options ); 188 | else 189 | [ EEG, com ] = eeg_eval( 'clean_artifacts', EEG, 'params', options ); 190 | end 191 | if fusechanrej 192 | if exist('pop_fusechanrej') 193 | EEG = pop_fusechanrej(EEG); 194 | else 195 | warning('Upgrade your version of EEGLAB to fuse channel rejection for datasets with same subject and session'); 196 | end 197 | end 198 | com = sprintf('EEG = pop_clean_rawdata(EEG, %s);', vararg2str(options)); 199 | return; 200 | end 201 | 202 | % Delete EEG.etc.clean_channel_mask and EEG.etc.clean_sample_mask if present. 203 | if isfield(EEG.etc, 'clean_channel_mask') 204 | EEG.etc = rmfield(EEG.etc, 'clean_channel_mask'); 205 | disp('EEG.etc.clean_channel_mask present: Deleted.') 206 | end 207 | if isfield(EEG.etc, 'clean_sample_mask') 208 | EEG.etc = rmfield(EEG.etc, 'clean_sample_mask'); 209 | disp('EEG.etc.clean_sample_mask present: Deleted.') 210 | end 211 | 212 | cleanEEG = clean_artifacts(EEG, options{:}); 213 | 214 | % Apply Christian's function before and after comparison visualization. 215 | if nargin < 2 && outs.vis == 1 216 | try 217 | vis_artifacts(cleanEEG,EEG); 218 | catch 219 | warning('vis_artifacts failed. Skipping visualization. Could be because of duplicate channel label.') 220 | end 221 | end 222 | 223 | % Update EEG. 224 | EEG = cleanEEG; 225 | 226 | % Output eegh. 227 | com = sprintf('EEG = pop_clean_rawdata(EEG, %s);', vararg2str(options)); 228 | 229 | % Display the ending message. 230 | disp('Done.') 231 | 232 | function outtext = wordwrap(intext,nChars) 233 | outtext = ''; 234 | while ~isempty(intext) 235 | if length(intext) > nChars 236 | cutoff = nChars+find([intext(nChars:end) ' ']==' ',1)-1; 237 | outtext = [outtext intext(1:cutoff-1) '\n']; %#ok<*AGROW> 238 | intext = intext(cutoff+1:end); 239 | else 240 | outtext = [outtext intext]; 241 | intext = ''; 242 | end 243 | end 244 | outtext = sprintf(outtext); 245 | -------------------------------------------------------------------------------- /private/block_geometric_median.m: -------------------------------------------------------------------------------- 1 | function y = block_geometric_median(X,blocksize,varargin) 2 | % Calculate a blockwise geometric median (faster and less memory-intensive 3 | % than the regular geom_median function). 4 | % 5 | % This statistic is not robust to artifacts that persist over a duration that 6 | % is significantly shorter than the blocksize. 7 | % 8 | % In: 9 | % X : the data (#observations x #variables) 10 | % blocksize : the number of successive samples over which a regular mean 11 | % should be taken 12 | % tol : tolerance (default: 1.e-5) 13 | % y : initial value (default: median(X)) 14 | % max_iter : max number of iterations (default: 500) 15 | % 16 | % Out: 17 | % g : geometric median over X 18 | % 19 | % Notes: 20 | % This function is noticably faster if the length of the data is divisible by the block size. 21 | % Uses the GPU if available. 22 | % 23 | 24 | % Copyright (C) Christian Kothe, SCCN, 2013, christian@sccn.ucsd.edu 25 | % 26 | % This program is free software; you can redistribute it and/or modify it under the terms of the GNU 27 | % General Public License as published by the Free Software Foundation; either version 2 of the 28 | % License, or (at your option) any later version. 29 | % 30 | % This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without 31 | % even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 32 | % General Public License for more details. 33 | % 34 | % You should have received a copy of the GNU General Public License along with this program; if not, 35 | % write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 36 | % USA 37 | 38 | if nargin < 2 || isempty(blocksize) 39 | blocksize = 1; end 40 | 41 | if blocksize > 1 42 | [o,v] = size(X); % #observations & #variables 43 | r = mod(o,blocksize); % #rest in last block 44 | b = (o-r)/blocksize; % #blocks 45 | if r > 0 46 | X = [reshape(sum(reshape(X(1:(o-r),:),blocksize,b*v)),b,v); sum(X((o-r+1):end,:))*(blocksize/r)]; 47 | else 48 | X = reshape(sum(reshape(X,blocksize,b*v)),b,v); 49 | end 50 | end 51 | 52 | try 53 | y = gather(geometric_median(gpuArray(X),varargin{:}))/blocksize; 54 | catch 55 | y = geometric_median(X,varargin{:})/blocksize; 56 | end 57 | -------------------------------------------------------------------------------- /private/design_fir.m: -------------------------------------------------------------------------------- 1 | function B = design_fir(N,F,A,nfft,W) 2 | % B = design_fir(N,F,A,nFFT,W) 3 | % Design an FIR filter using the frequency-sampling method. 4 | % 5 | % The frequency response is interpolated cubically between the specified 6 | % frequency points. 7 | % 8 | % In: 9 | % N : order of the filter 10 | % 11 | % F : vector of frequencies at which amplitudes shall be defined 12 | % (starts with 0 and goes up to 1; try to avoid too 13 | % sharp transitions) 14 | % 15 | % A : vector of amplitudes, one value per specified frequency 16 | % 17 | % nFFT : optionally number of FFT bins to use 18 | % 19 | % W : optionally the window function to use (default: Hamming) 20 | % 21 | % Out: 22 | % B : designed filter kernel 23 | % 24 | % Christian Kothe, Swartz Center for Computational Neuroscience, UCSD 25 | % 2013-08-14 26 | 27 | % Copyright (C) Christian Kothe, SCCN, 2013, ckothe@ucsd.edu 28 | % 29 | % This program is free software; you can redistribute it and/or modify it under the terms of the GNU 30 | % General Public License as published by the Free Software Foundation; either version 2 of the 31 | % License, or (at your option) any later version. 32 | % 33 | % This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without 34 | % even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 35 | % General Public License for more details. 36 | % 37 | % You should have received a copy of the GNU General Public License along with this program; if not, 38 | % write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 39 | % USA 40 | 41 | if nargin < 4 || isempty(nfft) 42 | nfft = max(512,2^ceil(log(N)/log(2))); end 43 | if nargin < 5 44 | W = 0.54 - 0.46*cos(2*pi*(0:N)/N); end 45 | 46 | % calculate interpolated frequency response 47 | F = interp1(round(F*nfft),A,(0:nfft),'pchip'); 48 | 49 | % set phase & transform into time domain 50 | F = F .* exp(-(0.5*N)*sqrt(-1)*pi*(0:nfft)./nfft); 51 | B = real(ifft([F conj(F(end-1:-1:2))])); 52 | 53 | % apply window to kernel 54 | B = B(1:N+1).*W(:)'; 55 | -------------------------------------------------------------------------------- /private/design_kaiser.m: -------------------------------------------------------------------------------- 1 | function W = design_kaiser(lo,hi,atten,odd) 2 | % Design a Kaiser window for a low-pass FIR filter 3 | % 4 | % In: 5 | % Lo : normalized lower frequency of transition band 6 | % 7 | % Hi : normalized upper frequency of transition band 8 | % 9 | % Attenuation : stop-band attenuation in dB (-20log10(ratio)) 10 | % 11 | % OddLength : whether the length shall be odd 12 | % 13 | % Out: 14 | % W : Designed window 15 | % 16 | % Christian Kothe, Swartz Center for Computational Neuroscience, UCSD 17 | % 2013-08-17 18 | 19 | % Copyright (C) Christian Kothe, SCCN, 2013, ckothe@ucsd.edu 20 | % 21 | % This program is free software; you can redistribute it and/or modify it under the terms of the GNU 22 | % General Public License as published by the Free Software Foundation; either version 2 of the 23 | % License, or (at your option) any later version. 24 | % 25 | % This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without 26 | % even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 27 | % General Public License for more details. 28 | % 29 | % You should have received a copy of the GNU General Public License along with this program; if not, 30 | % write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 31 | % USA 32 | 33 | % determine beta of the kaiser window 34 | if atten < 21 35 | beta = 0; 36 | elseif atten <= 50 37 | beta = 0.5842*(atten-21).^0.4 + 0.07886*(atten-21); 38 | else 39 | beta = 0.1102*(atten-8.7); 40 | end 41 | 42 | % determine the number of points 43 | N = round((atten-7.95)/(2*pi*2.285*(hi-lo)))+1; 44 | if odd && ~mod(N,2) 45 | N = N+1; end 46 | 47 | % design the window 48 | W = window_func('kaiser',N,beta); 49 | -------------------------------------------------------------------------------- /private/filter_fast.m: -------------------------------------------------------------------------------- 1 | function [X,Zf] = filter_fast(B,A,X,Zi,dim) 2 | % Like filter(), but faster when both the filter and the signal are long. 3 | % [Y,Zf] = filter_fast(B,A,X,Zi,Dim) 4 | % 5 | % Uses FFT convolution. The function is faster than filter when approx. length(B)>256 and 6 | % size(X,Dim)>1024, otherwise slower (due size-testing overhead). 7 | % 8 | % See also: 9 | % filter, fftfilt 10 | % 11 | % Christian Kothe, Swartz Center for Computational Neuroscience, UCSD 12 | % 2010-07-09 13 | % 14 | % contains fftfilt.m from Octave: 15 | % Copyright (C) 1996-1997 John W. Eaton 16 | 17 | 18 | % Copyright (C) Christian Kothe, SCCN, 2010, ckothe@ucsd.edu 19 | % 20 | % This program is free software; you can redistribute it and/or modify it under the terms of the GNU 21 | % General Public License as published by the Free Software Foundation; either version 2 of the 22 | % License, or (at your option) any later version. 23 | % 24 | % This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without 25 | % even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 26 | % General Public License for more details. 27 | % 28 | % You should have received a copy of the GNU General Public License along with this program; if not, 29 | % write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 30 | % USA 31 | 32 | 33 | if nargin <= 4 34 | dim = find(size(X)~=1,1); end 35 | if nargin <= 3 36 | Zi = []; end 37 | 38 | lenx = size(X,dim); 39 | lenb = length(B); 40 | if lenx == 0 41 | % empty X 42 | Zf = Zi; 43 | elseif lenb < 256 || lenx<1024 || lenx <= lenb || lenx*lenb < 4000000 || ~isequal(A,1) 44 | % use the regular filter 45 | if nargout > 1 46 | [X,Zf] = filter(B,A,X,Zi,dim); 47 | else 48 | X = filter(B,A,X,Zi,dim); 49 | end 50 | else 51 | was_single = strcmp(class(X),'single'); 52 | % fftfilt can be used 53 | if isempty(Zi) 54 | % no initial conditions to take care of 55 | if nargout < 2 56 | % and no final ones 57 | X = unflip(oct_fftfilt(B,flip(double(X),dim)),dim); 58 | else 59 | % final conditions needed 60 | X = flip(X,dim); 61 | [dummy,Zf] = filter(B,1,X(end-length(B)+1:end,:),Zi,1); %#ok 62 | X = oct_fftfilt(B,double(X)); 63 | X = unflip(X,dim); 64 | end 65 | else 66 | % initial conditions available 67 | X = flip(X,dim); 68 | % get a Zi-informed piece 69 | tmp = filter(B,1,X(1:length(B),:),Zi,1); 70 | if nargout > 1 71 | % also need final conditions 72 | [dummy,Zf] = filter(B,1,X(end-length(B)+1:end,:),Zi,1); %#ok 73 | end 74 | X = oct_fftfilt(B,double(X)); 75 | % incorporate the piece 76 | X(1:length(B),:) = tmp; 77 | X = unflip(X,dim); 78 | end 79 | if was_single 80 | X = single(X); end 81 | end 82 | 83 | function X = flip(X,dim) 84 | if dim ~= 1 85 | order = 1:ndims(X); 86 | order = order([dim 1]); 87 | X = permute(X,order); 88 | end 89 | 90 | function X = unflip(X,dim) 91 | if dim ~= 1 92 | order = 1:ndims(X); 93 | order = order([dim 1]); 94 | X = ipermute(X,order); 95 | end 96 | 97 | 98 | function y = oct_fftfilt(b, x, N) 99 | % Copyright (C) 1996, 1997 John W. Eaton 100 | % 101 | % This file is part of Octave. 102 | % 103 | % Octave is free software; you can redistribute it and/or modify it 104 | % under the terms of the GNU General Public License as published by 105 | % the Free Software Foundation; either version 2, or (at your option) 106 | % any later version. 107 | % 108 | % Octave is distributed in the hope that it will be useful, but 109 | % WITHOUT ANY WARRANTY; without even the implied warranty of 110 | % MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 111 | % General Public License for more details. 112 | % 113 | % You should have received a copy of the GNU General Public License 114 | % along with Octave; see the file COPYING. If not, write to the Free 115 | % Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 116 | % 02110-1301, USA. 117 | % 118 | % -*- texinfo -*- 119 | % @deftypefn {Function File} {} fftfilt (@var{b}, @var{x}, @var{n}) 120 | % 121 | % With two arguments, @code{fftfilt} filters @var{x} with the FIR filter 122 | % @var{b} using the FFT. 123 | % 124 | % Given the optional third argument, @var{n}, @code{fftfilt} uses the 125 | % overlap-add method to filter @var{x} with @var{b} using an N-point FFT. 126 | % 127 | % If @var{x} is a matrix, filter each column of the matrix. 128 | % @end deftypefn 129 | % 130 | % Author: Kurt Hornik 131 | % Created: 3 September 1994 132 | % Adapted-By: jwe 133 | 134 | % If N is not specified explicitly, we do not use the overlap-add 135 | % method at all because loops are really slow. Otherwise, we only 136 | % ensure that the number of points in the FFT is the smallest power 137 | % of two larger than N and length(b). This could result in length 138 | % one blocks, but if the user knows better ... 139 | transpose = (size(x,1) == 1); 140 | 141 | if transpose 142 | x = x.'; end 143 | 144 | [r_x,c_x] = size(x); 145 | [r_b,c_b] = size(b); 146 | if min([r_b, c_b]) ~= 1 147 | error('octave:fftfilt','fftfilt: b should be a vector'); end 148 | 149 | l_b = r_b*c_b; 150 | b = reshape(b,l_b,1); 151 | 152 | if nargin == 2 153 | % Use FFT with the smallest power of 2 which is >= length (x) + 154 | % length (b) - 1 as number of points ... 155 | N = 2^(ceil(log(r_x+l_b-1)/log(2))); 156 | B = fft(b,N); 157 | y = ifft(fft(x,N).*B(:,ones(1,c_x))); 158 | else 159 | % Use overlap-add method ... 160 | if ~isscalar(N) 161 | error ('octave:fftfilt','fftfilt: N has to be a scalar'); end 162 | N = 2^(ceil(log(max([N,l_b]))/log(2))); 163 | L = N - l_b + 1; 164 | B = fft(b, N); 165 | B = B(:,ones(c_x,1)); 166 | R = ceil(r_x / L); 167 | y = zeros(r_x, c_x); 168 | for r = 1:R 169 | lo = (r - 1) * L + 1; 170 | hi = min(r * L, r_x); 171 | tmp = zeros(N, c_x); 172 | tmp(1:(hi-lo+1),:) = x(lo:hi,:); 173 | tmp = ifft(fft(tmp).*B); 174 | hi = min(lo+N-1, r_x); 175 | y(lo:hi,:) = y(lo:hi,:) + tmp(1:(hi-lo+1),:); 176 | end 177 | end 178 | 179 | y = y(1:r_x,:); 180 | if transpose 181 | y = y.'; end 182 | 183 | % Final cleanups: if both x and b are real respectively integer, y 184 | % should also be 185 | if isreal(b) && isreal(x) 186 | y = real(y); end 187 | if ~any(b - round(b)) 188 | idx = ~any(x - round(x)); 189 | y(:,idx) = round(y(:,idx)); 190 | end 191 | -------------------------------------------------------------------------------- /private/filtfilt_fast.m: -------------------------------------------------------------------------------- 1 | function X = filtfilt_fast(varargin) 2 | % Like filtfilt(), but faster when filter and signal are long (and A=1). 3 | % Y = filtfilt_fast(B,A,X) 4 | % 5 | % Uses FFT convolution (needs fftfilt). The function is faster than filter when approx. 6 | % length(B)>256 and size(X,Dim)>1024, otherwise slower (due size-testing overhead). 7 | % 8 | % Note: 9 | % Can also be called with four arguments, as Y = filtfilt_fast(N,F,A,X), in which case an Nth order 10 | % FIR filter is designed that has the desired frequency response A at normalized frequencies F; F 11 | % must be a vector of numbers increasing from 0 to 1. 12 | % 13 | % See also: 14 | % filtfilt, filter 15 | % 16 | % Christian Kothe, Swartz Center for Computational Neuroscience, UCSD 17 | % 2010-07-14 18 | 19 | % Copyright (C) Christian Kothe, SCCN, 2010, ckothe@ucsd.edu 20 | % 21 | % This program is free software; you can redistribute it and/or modify it under the terms of the GNU 22 | % General Public License as published by the Free Software Foundation; either version 2 of the 23 | % License, or (at your option) any later version. 24 | % 25 | % This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without 26 | % even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 27 | % General Public License for more details. 28 | % 29 | % You should have received a copy of the GNU General Public License along with this program; if not, 30 | % write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 31 | % USA 32 | 33 | if nargin == 3 34 | [B A X] = deal(varargin{:}); 35 | elseif nargin == 4 36 | [N F M X] = deal(varargin{:}); 37 | B = design_fir(N,F,sqrt(M)); A = 1; % note: we use the sqrt() because we run forward and backward 38 | else 39 | help filtfilt_fast; 40 | return; 41 | end 42 | 43 | if A == 1 44 | was_single = strcmp(class(X),'single'); 45 | w = length(B); t = size(X,1); 46 | % extrapolate 47 | X = double([bsxfun(@minus,2*X(1,:),X(1+mod(((w+1):-1:2)-1,t),:)); X; bsxfun(@minus,2*X(t,:),X(1+mod(((t-1):-1:(t-w))-1,t),:))]); 48 | % filter, reverse 49 | X = filter_fast(B,A,X); X = X(length(X):-1:1,:); 50 | % filter, reverse 51 | X = filter_fast(B,A,X); X = X(length(X):-1:1,:); 52 | % remove extrapolated pieces 53 | X([1:w t+w+(1:w)],:) = []; 54 | if was_single 55 | X = single(X); end 56 | else 57 | % fall back to filtfilt for the IIR case 58 | X = filtfilt(B,A,X); 59 | end 60 | -------------------------------------------------------------------------------- /private/fit_eeg_distribution.m: -------------------------------------------------------------------------------- 1 | function [mu,sig,alpha,beta] = fit_eeg_distribution(X,min_clean_fraction,max_dropout_fraction,quants,step_sizes,beta) 2 | % Estimate the mean and standard deviation of clean EEG from contaminated data. 3 | % [Mu,Sigma,Alpha,Beta] = fit_eeg_distribution(X,MinCleanFraction,MaxDropoutFraction,FitQuantiles,StepSizes,ShapeRange) 4 | % 5 | % This function estimates the mean and standard deviation of clean EEG from a sample of amplitude 6 | % values (that have preferably been computed over short windows) that may include a large fraction 7 | % of contaminated samples. The clean EEG is assumed to represent a generalized Gaussian component in 8 | % a mixture with near-arbitrary artifact components. By default, at least 25% (MinCleanFraction) of 9 | % the data must be clean EEG, and the rest can be contaminated. No more than 10% 10 | % (MaxDropoutFraction) of the data is allowed to come from contaminations that cause lower-than-EEG 11 | % amplitudes (e.g., sensor unplugged). There are no restrictions on artifacts causing 12 | % larger-than-EEG amplitudes, i.e., virtually anything is handled (with the exception of a very 13 | % unlikely type of distribution that combines with the clean EEG samples into a larger symmetric 14 | % generalized Gaussian peak and thereby "fools" the estimator). The default parameters should be 15 | % fine for a wide range of settings but may be adapted to accomodate special circumstances. 16 | % 17 | % The method works by fitting a truncated generalized Gaussian whose parameters are constrained by 18 | % MinCleanFraction, MaxDropoutFraction, FitQuantiles, and ShapeRange. The alpha and beta parameters 19 | % of the gen. Gaussian are also returned. The fit is performed by a grid search that always finds a 20 | % close-to-optimal solution if the above assumptions are fulfilled. 21 | % 22 | % In: 23 | % X : vector of amplitude values of EEG, possible containing artifacts 24 | % (coming from single samples or windowed averages) 25 | % 26 | % MinCleanFraction : Minimum fraction of values in X that needs to be clean 27 | % (default: 0.25) 28 | % 29 | % MaxDropoutFraction : Maximum fraction of values in X that can be subject to 30 | % signal dropouts (e.g., sensor unplugged) (default: 0.1) 31 | % 32 | % FitQuantiles : Quantile range [lower,upper] of the truncated generalized Gaussian distribution 33 | % that shall be fit to the EEG contents (default: [0.022 0.6]) 34 | % 35 | % StepSizes : Step size of the grid search; the first value is the stepping of the lower bound 36 | % (which essentially steps over any dropout samples), and the second value 37 | % is the stepping over possible scales (i.e., clean-data quantiles) 38 | % (default: [0.01 0.01]) 39 | % 40 | % ShapeRange : Range that the clean EEG distribution's shape parameter beta may take (default: 41 | % 1.7:0.15:3.5) 42 | % 43 | % Out: 44 | % Mu : estimated mean of the clean EEG distribution 45 | % 46 | % Sigma : estimated standard deviation of the clean EEG distribution 47 | % 48 | % Alpha : estimated scale parameter of the generalized Gaussian clean EEG distribution (optional) 49 | % 50 | % Beta : estimated shape parameter of the generalized Gaussian clean EEG distribution (optional) 51 | % 52 | % Christian Kothe, Swartz Center for Computational Neuroscience, UCSD 53 | % 2013-08-15 54 | 55 | % Copyright (C) Christian Kothe, SCCN, 2013, ckothe@ucsd.edu 56 | % 57 | % This program is free software; you can redistribute it and/or modify it under the terms of the GNU 58 | % General Public License as published by the Free Software Foundation; either version 2 of the 59 | % License, or (at your option) any later version. 60 | % 61 | % This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without 62 | % even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 63 | % General Public License for more details. 64 | % 65 | % You should have received a copy of the GNU General Public License along with this program; if not, 66 | % write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 67 | % USA 68 | 69 | % assign defaults 70 | if ~exist('min_clean_fraction','var') || isempty(min_clean_fraction) 71 | min_clean_fraction = 0.25; end 72 | if ~exist('max_dropout_fraction','var') || isempty(max_dropout_fraction) 73 | max_dropout_fraction = 0.1; end 74 | if ~exist('quants','var') || isempty(quants) 75 | quants = [0.022 0.6]; end 76 | if ~exist('step_sizes','var') || isempty(step_sizes) 77 | step_sizes = [0.01 0.01]; end 78 | if ~exist('beta','var') || isempty(beta) 79 | beta = 1.7:0.15:3.5; end 80 | 81 | % sanity checks 82 | if ~isvector(quants) || numel(quants) > 2 83 | error('Fit quantiles needs to be a 2-element vector (support for matrices deprecated).'); end 84 | if any(quants(:)<0) || any(quants(:)>1) 85 | error('Unreasonable fit quantiles.'); end 86 | if any(step_sizes<0.0001) || any(step_sizes>0.1) 87 | error('Unreasonable step sizes.'); end 88 | if any(beta>=7) || any(beta<=1) 89 | error('Unreasonable shape range.'); end 90 | 91 | % sort data so we can access quantiles directly 92 | X = double(sort(X(:))); 93 | n = length(X); 94 | 95 | % calc z bounds for the truncated standard generalized Gaussian pdf and pdf rescaler 96 | for b=1:length(beta) 97 | zbounds{b} = sign(quants-1/2).*gammaincinv(sign(quants-1/2).*(2*quants-1),1/beta(b)).^(1/beta(b)); %#ok<*AGROW> 98 | rescale(b) = beta(b)/(2*gamma(1/beta(b))); 99 | end 100 | 101 | % determine the quantile-dependent limits for the grid search 102 | lower_min = min(quants); % we can generally skip the tail below the lower quantile 103 | max_width = diff(quants); % maximum width is the fit interval if all data is clean 104 | min_width = min_clean_fraction*max_width; % minimum width of the fit interval, as fraction of data 105 | 106 | % get matrix of shifted data ranges 107 | X = X(bsxfun(@plus,(1:round(n*max_width))',round(n*(lower_min:step_sizes(1):lower_min+max_dropout_fraction)))); 108 | X1 = X(1,:); X = bsxfun(@minus,X,X1); 109 | 110 | opt_val = Inf; 111 | % for each interval width... 112 | for m = round(n*(max_width:-step_sizes(2):min_width)) 113 | % scale and bin the data in the intervals 114 | nbins = round(3*log2(1+m/2)); 115 | H = bsxfun(@times,X(1:m,:),nbins./X(m,:)); 116 | logq = log(histc(H,[0:nbins-1,Inf]) + 0.01); 117 | 118 | % for each shape value... 119 | for b=1:length(beta) 120 | bounds = zbounds{b}; 121 | % evaluate truncated generalized Gaussian pdf at bin centers 122 | x = bounds(1)+(0.5:(nbins-0.5))/nbins*diff(bounds); 123 | p = exp(-abs(x).^beta(b))*rescale(b); p=p'/sum(p); 124 | 125 | % calc KL divergences 126 | kl = sum(bsxfun(@times,p,bsxfun(@minus,log(p),logq(1:end-1,:)))) + log(m); 127 | 128 | % update optimal parameters 129 | [min_val,idx] = min(kl); 130 | if min_val < opt_val 131 | opt_val = min_val; 132 | opt_beta = beta(b); 133 | opt_bounds = bounds; 134 | opt_lu = [X1(idx) X1(idx)+X(m,idx)]; 135 | end 136 | end 137 | end 138 | 139 | % recover distribution parameters at optimum 140 | alpha = (opt_lu(2)-opt_lu(1))/diff(opt_bounds); 141 | mu = opt_lu(1)-opt_bounds(1)*alpha; 142 | beta = opt_beta; 143 | 144 | % calculate the distribution's standard deviation from alpha and beta 145 | sig = sqrt((alpha^2)*gamma(3/beta)/gamma(1/beta)); 146 | -------------------------------------------------------------------------------- /private/geometric_median.m: -------------------------------------------------------------------------------- 1 | function y = geometric_median(X,tol,y,max_iter) 2 | % Calculate the geometric median for a set of observations (mean under a Laplacian noise distribution) 3 | % This is using Weiszfeld's algorithm. 4 | % 5 | % In: 6 | % X : the data, as in mean 7 | % tol : tolerance (default: 1.e-5) 8 | % y : initial value (default: median(X)) 9 | % max_iter : max number of iterations (default: 500) 10 | % 11 | % Out: 12 | % g : geometric median over X 13 | 14 | % Copyright (C) Christian Kothe, SCCN, 2012, ckothe@ucsd.edu 15 | % 16 | % This program is free software; you can redistribute it and/or modify it under the terms of the GNU 17 | % General Public License as published by the Free Software Foundation; either version 2 of the 18 | % License, or (at your option) any later version. 19 | % 20 | % This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without 21 | % even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 22 | % General Public License for more details. 23 | % 24 | % You should have received a copy of the GNU General Public License along with this program; if not, 25 | % write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 26 | % USA 27 | 28 | if ~exist('tol','var') || isempty(tol) 29 | tol = 1.e-5; end 30 | if ~exist('y','var') || isempty(y) 31 | y = median(X); end 32 | if ~exist('max_iter','var') || isempty(max_iter) 33 | max_iter = 500; end 34 | 35 | for i=1:max_iter 36 | invnorms = 1./sqrt(sum(bsxfun(@minus,X,y).^2,2)); 37 | [y,oldy] = deal(sum(bsxfun(@times,X,invnorms)) / sum(invnorms),y); 38 | if norm(y-oldy)/norm(y) < tol 39 | break; end 40 | end 41 | -------------------------------------------------------------------------------- /private/hlp_handleerror.m: -------------------------------------------------------------------------------- 1 | function s = hlp_handleerror(e,level) 2 | % Displays a formatted error message for some error object, including a full stack trace. 3 | % hlp_handleerror(Error, Indent) 4 | % 5 | % In: 6 | % Error : error object, as received from lasterror or via a catch clause 7 | % (if omitted, lasterror is used) 8 | % Indent : optional indentation level, in characters 9 | % 10 | % Out: 11 | % Formatted : optionally the formatted error report 12 | % 13 | % Example: 14 | % % display an error message including stack trace in case of an error and continue without 15 | % % terminating 16 | % try 17 | % ... 18 | % catch e 19 | % hlp_handleerror(e); 20 | % end 21 | % 22 | % See also: 23 | % lasterror, MException, catch 24 | % 25 | % Christian Kothe, Swartz Center for Computational Neuroscience, UCSD 26 | % 2010-04-22 27 | 28 | % Copyright (C) Christian Kothe, SCCN, 2010, ckothe@ucsd.edu 29 | % 30 | % This program is free software; you can redistribute it and/or modify it under the terms of the GNU 31 | % General Public License as published by the Free Software Foundation; either version 2 of the 32 | % License, or (at your option) any later version. 33 | % 34 | % This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without 35 | % even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 36 | % General Public License for more details. 37 | % 38 | % You should have received a copy of the GNU General Public License along with this program; if not, 39 | % write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 40 | % USA 41 | 42 | try 43 | if ~exist('e','var') 44 | e = lasterror; end %#ok 45 | 46 | % compute the appropriate indentation level 47 | if ~exist('level','var') 48 | level = ''; 49 | else 50 | level = repmat(' ',1,level); 51 | end 52 | 53 | if nargout == 0 54 | % display the message 55 | for message = hlp_split(e.message,[10 13]) 56 | fprintf('%s %s\n',level,message{1}); end 57 | fprintf('%s occurred in: \n',level); 58 | for st = e.stack' 59 | if ~isdeployed 60 | try 61 | fprintf('%s %s: %i\n',level,st.file,st.line,st.name,st.line); 62 | catch 63 | fprintf('%s %s: %i\n',level,st.file,st.name,st.line); 64 | end 65 | else 66 | % links are not supported in deployed mode 67 | fprintf('%s %s: %i\n',level,st.file,st.line); 68 | end 69 | end 70 | else 71 | % print message into a string 72 | s = []; 73 | for message = hlp_split(e.message,[10 13]) 74 | s = [s sprintf('%s %s\n',level,message{1})]; end 75 | s = [s sprintf('%s occurred in: \n',level)]; 76 | for st = e.stack' 77 | s = [s sprintf('%s %s: %i\n',level,st.file,st.line)]; end 78 | end 79 | catch 80 | disp('An error occurred, but the traceback could not be displayed due to another error...'); 81 | end 82 | -------------------------------------------------------------------------------- /private/hlp_microcache.m: -------------------------------------------------------------------------------- 1 | function varargout = hlp_microcache(dom, f, varargin) 2 | % Cache results of functions for repeated calls with the same arguments. 3 | % Results... = hlp_microcache(Domain, Function, Arguments...) 4 | % 5 | % This is a lightweight mechanism to memoize results of functions that are often repeatedly called 6 | % with the same arguments. It is indended for small-scale situations -- i.e., the function is called 7 | % only with a small variety of arguments, for example less than 100, and the arguments (as well as 8 | % results) are not too big (e.g., no large matrices or data sets). If too many different calls are 9 | % to be cached, hlp_microcache "forgets" the oldest ones. Its memory is also lost after MATLAB is 10 | % closed or after typing "clear all". Different places of a program can independently memoize 11 | % results of their functions, by calling hlp_microcache with their own unique 'domain' identifier. 12 | % The overhead is comparable to that of MATLAB's num2str(), i.e., very low. 13 | % 14 | % In: 15 | % Domain : arbitrary (MATLAB-conformant) string identifier of the cache 'domain' 16 | % may be used to keep separate matters separate from each other 17 | % 18 | % Function : function handle to compute a result from some arguments 19 | % 20 | % Arguments... : arguments to pass to the function 21 | % 22 | % Out: 23 | % Results... : return values of the function for the given arguments 24 | % 25 | % 26 | % Notes: 27 | % Only *referentially transparent* functions are allowed; if a function can give different outputs 28 | % for the same arguments, this can lead to subtle bugs; examples include functions that refer to 29 | % global state. This can be fixed by turning all dependencies of the function into arguments. 30 | % 31 | % A special note applies to lambda functions in the function's arguments (even deeply nested). 32 | % Every instance of such a lambda function is usually considered unique by MATLAB because lambdas 33 | % implicitly reference all variables in the scope where they were created, so hlp_microcache will 34 | % re-invoke the function eagerly. To ameliorate this, top-level lambda terms in the arguments can 35 | % be treated differently if the flag 'lambda_equality' is set to 'proper' or 'string' for the domain 36 | % (see Advanced section). 37 | % 38 | % Special care has to be taken if a function behaves differently depending on the types of 39 | % arguments, especially uint and int types (which behave distinctly differently than doubles in 40 | % calculations): If the input values are the same numbers as for a previously cached result, 41 | % hlp_microcache will return that result regardless of the data type! You can work aroun this 42 | % where necessary by also passing the type of the affected arguments as to your function (this 43 | % way, matching will be type-sensitive) 44 | % 45 | % This function is safe for concurrent use via timers. 46 | % 47 | % Examples: 48 | % % if this line is executed for the first time, it is as slow as magic(2000) 49 | % m = hlp_microcache('test',@magic,2000); 50 | % 51 | % % if it is executed a second time, it is three orders of magnitude faster than m=magic(2000) 52 | % m = hlp_microcache('test',@magic,2000); 53 | % 54 | % 55 | % Advanced: 56 | % When called as hlp_microcache(Domain, 'option1', value1, 'option2', value2, ...), the options 57 | % override aspects of the caching policy for the respective domain. Possible options are: 58 | % 'resort_freq' : re-sort cached argument sets by their usage frequency every resort_freq lookups (default: 10) 59 | % note: the number is not per domain but per group of equally-sized argument sets. 60 | % 'group_size' : retain only the group_size most-recently used argument sets (default: 100) 61 | % note: the number is not per domain but per group of equally-sized argument sets. 62 | % 'max_key_size': maximum size of keys that are cached, in bytes (larger ones are not cached) (default: 100000000) 63 | % 'max_result_size': maximum size of results that are cached, in bytes (larger ones are not cached) (default: 100000000) 64 | % 'lambda_equality': how the equality of lambda functions (that are not nested into data structures) is determined 65 | % - false: the default MATLAB behavior will be used (fastest); this will consider 66 | % lambda's to be non-equal in far more cases than strictly necessary, 67 | % because they automatically bind all variables in the same scope as 68 | % where they were created (default) 69 | % - 'proper': the 'proper' way to handle top-level lambda functions with bound 70 | % arguments: any non-referenced variables from the same scope of 71 | % the lambda function will be excluded from comparison (which works 72 | % except for corner cases where the lambda term refers to them 73 | % using expressions like eval()) 74 | % - 'string': in this case, only the string representation will be compared -- 75 | % ignoring any directly or indirectly bound variables (which is 76 | % fast); this works for simple expressions, but *will* give 77 | % incorrect results in more complicated cases, including 78 | % expressions such as @self.method (which MATLAB implicitly turns 79 | % into a lambda term!). 80 | % 81 | % When called as hlp_microcache(Domain, 'clear'), the cached contents of the domain will be cleared, 82 | % and when called as hlp_microcache('clear'), the contents of *all* domains are cleared. Instead of 83 | % 'clear', 'reset' can be used, which in addition resets the configurations for the affected domains. 84 | % 85 | % Advanced Examples: 86 | % % for the caching domain 'test', set a different resorting frequency and a different group size 87 | % hlp_microcache('test', 'resort_freq',30, 'group_size',100); 88 | % 89 | % % clear the caching domain 'test' (configurations remain) 90 | % hlp_microcache('test', 'clear'); 91 | % 92 | % % reset the caching domain 'test' (including configurations) 93 | % hlp_microcache('test', 'reset'); 94 | % 95 | % % clear the contents of all caching domains (configurations remain) 96 | % hlp_microcache('clear'); 97 | % 98 | % % reset the micro-cache (including configuration) 99 | % hlp_microcache('reset'); 100 | % 101 | % Christian Kothe, Swartz Center for Computational Neuroscience, UCSD 102 | % 2010-06-18 103 | 104 | % Copyright (C) Christian Kothe, SCCN, 2010, christian@sccn.ucsd.edu 105 | % 106 | % This program is free software; you can redistribute it and/or modify it under the terms of the GNU 107 | % General Public License as published by the Free Software Foundation; either version 2 of the 108 | % License, or (at your option) any later version. 109 | % 110 | % This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without 111 | % even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 112 | % General Public License for more details. 113 | % 114 | % You should have received a copy of the GNU General Public License along with this program; if not, 115 | % write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 116 | % USA 117 | 118 | 119 | % mc ("microcache"): struct 120 | % domain-id --> domainpool 121 | % domainpool: struct 122 | % config --> configuration 123 | % size-id --> sizepool 124 | % sizepool: struct 125 | % outs: cell array of {f's outputs} 126 | % inps: cell array of [{char(f)},varargin,{nargout}] 127 | % frqs: double array of usage frequencies 128 | % luse: double array of last use cputime 129 | % lcnt: lookup count for this pool; used to schedule resorting (by usage frequency) 130 | % configuration: struct 131 | % resort_freq: re-sort a sizepool every this many lookups (into that pool) 132 | % group_size: maximum number of entries per size-pool 133 | % max_result_size: maximum size of results that are cached 134 | % max_key_size: maximum size of keys that are cached 135 | % lambda_equality: desired equality relation for lambda expressions 136 | persistent mc; 137 | 138 | % [varargout{1:nargout}] = f(varargin{:}); return; % UNCOMMENT TO BYPASS 139 | 140 | % is this a regular call? 141 | if nargin > 1 142 | if isa(f,'function_handle') 143 | 144 | % get the current lookup time 145 | now = cputime; 146 | 147 | % compute the key structure 148 | cf = char(f); 149 | if cf(1) == '@' 150 | % ignore variables only accessible via eval() / evalin() in lambda terms 151 | tmp = functions(f); 152 | key_f = {cf,tmp.workspace{1}}; 153 | else 154 | key_f = f; 155 | end 156 | 157 | try 158 | % optionally do some special processing of lambda terms in the function arguments 159 | args = varargin; 160 | leq = mc.(dom).config.lambda_equality; 161 | if ischar(leq) 162 | for k = find(cellfun('isclass',varargin,'function_handle')) 163 | ck = char(args{k}); 164 | if ck(1) == '@' 165 | if strcmp(leq,'string') 166 | % the fastest way to deal with them is to scrap any possible bound variables 167 | % and just consider the raw expression 168 | args{k} = ck; 169 | elseif strcmp(leq,'proper') 170 | % the 'proper' way for most purposes is to consider both the expression and 171 | % the directly bound variables but ignore variables in the same scope 172 | tmp = functions(args{k}); 173 | args{k} = {ck,tmp.workspace{1}}; 174 | else 175 | disp('hlp_microcache: unrecognized lambda_equality setting encountered; resetting to false.'); 176 | end 177 | end 178 | end 179 | end 180 | catch %#ok 181 | % the lambda lookup failed, so we assume that it has not yet been assigned; assign it now. 182 | mc.(dom).config.lambda_equality = false; 183 | end 184 | key = [args,{key_f,nargout}]; 185 | 186 | % get the size id (sid) of the key (MATLAB keeps track of that for every object) 187 | keyinfo = whos('key'); 188 | keysid = sprintf('s%u',keyinfo.bytes); 189 | 190 | try 191 | % retrieve the pool of size-equivalent objects 192 | sizepool = mc.(dom).(keysid); 193 | % search for the key in the pool (checking the most-frequently used keys first) 194 | for k=1:length(sizepool.inps) 195 | if isequalwithequalnans(key,sizepool.inps{k}) %#ok % (isequalwithequalnans() is a fast builtin) 196 | % found the key, deliver outputs 197 | varargout = sizepool.outs{k}; 198 | % update the db record... 199 | sizepool.frqs(k) = sizepool.frqs(k)+1; 200 | sizepool.luse(k) = now; 201 | sizepool.lcnt = sizepool.lcnt+1; 202 | % resort by lookup frequency every resort_freq lookups 203 | if sizepool.lcnt > mc.(dom).config.resort_freq 204 | [sizepool.frqs,inds] = sort(sizepool.frqs,'descend'); 205 | sizepool.inps = sizepool.inps(inds); 206 | sizepool.outs = sizepool.outs(inds); 207 | sizepool.luse = sizepool.luse(inds); 208 | sizepool.lcnt = 0; 209 | end 210 | % write back 211 | mc.(dom).(keysid) = sizepool; 212 | return; 213 | end 214 | end 215 | catch %#ok 216 | % domain+keysid not yet in the cache: create appropriate structures (this is rare) 217 | sizepool = struct('inps',{{}}, 'outs',{{}}, 'frqs',{[]}, 'luse',{[]}, 'lcnt',{0}); 218 | end 219 | 220 | if ~exist('varargout','var') 221 | % set up the default configuration for a domain, if it's not yet present 222 | if ~isfield(mc,dom) || ~isfield(mc.(dom),'config') 223 | mc.(dom).config = struct(); end 224 | if ~isfield(mc.(dom).config,'resort_freq') 225 | mc.(dom).config.resort_freq = 10; end 226 | if ~isfield(mc.(dom).config,'group_size') 227 | mc.(dom).config.group_size = 100; end 228 | if ~isfield(mc.(dom).config,'max_result_size') 229 | mc.(dom).config.max_result_size = 100000000; end 230 | if ~isfield(mc.(dom).config,'max_key_size') 231 | mc.(dom).config.max_key_size = 100000000; end 232 | % did not find the entry in the size pool: compute it 233 | [varargout{1:nargout}] = f(varargin{:}); 234 | iteminfo = whos('varargout'); 235 | if iteminfo.bytes <= mc.(dom).config.max_result_size && keyinfo.bytes <= mc.(dom).config.max_key_size 236 | % add to pool 237 | sizepool.luse(end+1) = now; 238 | sizepool.frqs(end+1) = 1; 239 | sizepool.inps{end+1} = key; 240 | sizepool.outs{end+1} = varargout; 241 | sizepool.lcnt = 0; 242 | % remove least-recently used entries if necessary 243 | while length(sizepool.inps) > mc.(dom).config.group_size 244 | [x,idx] = min(sizepool.luse); %#ok 245 | sizepool.luse(idx) = []; 246 | sizepool.frqs(idx) = []; 247 | sizepool.inps(idx) = []; 248 | sizepool.outs(idx) = []; 249 | end 250 | % write back 251 | mc.(dom).(keysid) = sizepool; 252 | end 253 | end 254 | 255 | else 256 | % a control invocation 257 | varargin = [{f} varargin]; 258 | if length(varargin) > 1 259 | % f and what follows are name-value pairs that define the domain configuration 260 | for k=1:2:length(varargin) 261 | mc.(dom).config.(varargin{k}) = varargin{k+1}; end 262 | else 263 | % single argument: this is a command to be applied to the domain 264 | switch varargin{1} 265 | case 'clear' 266 | % clear the entire domain 267 | if isfield(mc,dom) 268 | if isfield(mc.(dom),'config') 269 | % retain the config if applicable 270 | mc.(dom) = struct('config',{mc.(dom).config}); 271 | else 272 | mc = rmfield(mc,dom); 273 | end 274 | end 275 | case 'reset' 276 | % reset the domain, including config 277 | mc = rmfield(mc,dom); 278 | otherwise 279 | error('Unrecognized command.'); 280 | end 281 | end 282 | end 283 | else 284 | % global command (no f specified) 285 | if strcmp(dom,'clear') && isstruct(mc) 286 | % clear all domains... 287 | for d = fieldnames(mc)' 288 | hlp_microcache(d{1},'clear'); end 289 | elseif strcmp(dom,'reset') 290 | % reset all domains 291 | mc = []; 292 | end 293 | end 294 | 295 | -------------------------------------------------------------------------------- /private/hlp_split.m: -------------------------------------------------------------------------------- 1 | function res = hlp_split(str,delims) 2 | % Split a string according to some delimiter(s). 3 | % Result = hlp_split(String,Delimiters) 4 | % 5 | % In: 6 | % String : a string (char vector) 7 | % 8 | % Delimiters : a vector of delimiter characters (includes no special support for escape sequences) 9 | % 10 | % Out: 11 | % Result : a cell array of (non-empty) non-Delimiter substrings in String 12 | % 13 | % Examples: 14 | % % split a string at colons and semicolons; returns a cell array of four parts 15 | % hlp_split('sdfdf:sdfsdf;sfdsf;;:sdfsdf:',':;') 16 | % 17 | % Christian Kothe, Swartz Center for Computational Neuroscience, UCSD 18 | % 2010-11-05 19 | 20 | % Copyright (C) Christian Kothe, SCCN, 2010, ckothe@ucsd.edu 21 | % 22 | % This program is free software; you can redistribute it and/or modify it under the terms of the GNU 23 | % General Public License as published by the Free Software Foundation; either version 2 of the 24 | % License, or (at your option) any later version. 25 | % 26 | % This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without 27 | % even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 28 | % General Public License for more details. 29 | % 30 | % You should have received a copy of the GNU General Public License along with this program; if not, 31 | % write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 32 | % USA 33 | 34 | pos = find(diff([0 ~sum(bsxfun(@eq,str(:)',delims(:)),1) 0])); 35 | res = cell(~isempty(pos),length(pos)/2); 36 | for k=1:length(res) 37 | res{k} = str(pos(k*2-1):pos(k*2)-1); end 38 | -------------------------------------------------------------------------------- /private/hlp_varargin2struct.m: -------------------------------------------------------------------------------- 1 | function res = hlp_varargin2struct(args, varargin) 2 | % Convert a list of name-value pairs into a struct with values assigned to names. 3 | % struct = hlp_varargin2struct(Varargin, Defaults) 4 | % 5 | % In: 6 | % Varargin : cell array of name-value pairs and/or structs (with values assigned to names) 7 | % 8 | % Defaults : optional list of name-value pairs, encoding defaults; multiple alternative names may 9 | % be specified in a cell array 10 | % 11 | % Example: 12 | % function myfunc(x,y,z,varargin) 13 | % % parse options, and give defaults for some of them: 14 | % options = hlp_varargin2struct(varargin, 'somearg',10, 'anotherarg',{1 2 3}); 15 | % 16 | % Notes: 17 | % * mandatory args can be expressed by specifying them as ..., 'myparam',mandatory, ... in the defaults 18 | % an error is raised when any of those is left unspecified 19 | % 20 | % * the following two parameter lists are equivalent (note that the struct is specified where a name would be expected, 21 | % and that it replaces the entire name-value pair): 22 | % ..., 'xyz',5, 'a',[], 'test','toast', 'xxx',{1}. ... 23 | % ..., 'xyz',5, struct( 'a',{[]},'test',{'toast'} ), 'xxx',{1}, ... 24 | % 25 | % * names with dots are allowed, i.e.: ..., 'abc',5, 'xxx.a',10, 'xxx.yyy',20, ... 26 | % 27 | % * some parameters may have multiple alternative names, which shall be remapped to the 28 | % standard name within opts; alternative names are given together with the defaults, 29 | % by specifying a cell array of names instead of the name in the defaults, as in the following example: 30 | % ... ,{'standard_name','alt_name_x','alt_name_y'}, 20, ... 31 | % 32 | % Out: 33 | % Result : a struct with fields corresponding to the passed arguments (plus the defaults that were 34 | % not overridden); if the caller function does not retrieve the struct, the variables are 35 | % instead copied into the caller's workspace. 36 | % 37 | % Examples: 38 | % % define a function which takes some of its arguments as name-value pairs 39 | % function myfunction(myarg1,myarg2,varargin) 40 | % opts = hlp_varargin2struct(varargin, 'myarg3',10, 'myarg4',1001, 'myarg5','test'); 41 | % 42 | % % as before, but this time allow an alternative name for myarg3 43 | % function myfunction(myarg1,myarg2,varargin) 44 | % opts = hlp_varargin2struct(varargin, {'myarg3','legacyargXY'},10, 'myarg4',1001, 'myarg5','test'); 45 | % 46 | % % as before, but this time do not return arguments in a struct, but assign them directly to the 47 | % % function's workspace 48 | % function myfunction(myarg1,myarg2,varargin) 49 | % hlp_varargin2struct(varargin, {'myarg3','legacyargXY'},10, 'myarg4',1001, 'myarg5','test'); 50 | % 51 | % See also: 52 | % hlp_struct2varargin, arg_define 53 | % 54 | % Christian Kothe, Swartz Center for Computational Neuroscience, UCSD 55 | % 2010-04-05 56 | 57 | % Copyright (C) Christian Kothe, SCCN, 2010, ckothe@ucsd.edu 58 | % 59 | % This program is free software; you can redistribute it and/or modify it under the terms of the GNU 60 | % General Public License as published by the Free Software Foundation; either version 2 of the 61 | % License, or (at your option) any later version. 62 | % 63 | % This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without 64 | % even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 65 | % General Public License for more details. 66 | % 67 | % You should have received a copy of the GNU General Public License along with this program; if not, 68 | % write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 69 | % USA 70 | 71 | if isstruct(args) 72 | args = {args}; end 73 | 74 | % --- handle defaults --- 75 | if ~isempty(varargin) 76 | % splice substructs into the name-value list 77 | if any(cellfun('isclass',varargin(1:2:end),'struct')) 78 | varargin = flatten_structs(varargin); end 79 | 80 | defnames = varargin(1:2:end); 81 | defvalues = varargin(2:2:end); 82 | 83 | % make a remapping table for alternative default names... 84 | for k=find(cellfun('isclass',defnames,'cell')) 85 | for l=2:length(defnames{k}) 86 | name_for_alternative.(defnames{k}{l}) = defnames{k}{1}; end 87 | defnames{k} = defnames{k}{1}; 88 | end 89 | 90 | % create default struct 91 | if [defnames{:}]~='.' 92 | % use only the last assignment for each name 93 | [s,indices] = sort(defnames(:)); 94 | indices( strcmp(s((1:end-1)'),s((2:end)'))) = []; 95 | % and make the struct 96 | res = cell2struct(defvalues(indices),defnames(indices),2); 97 | else 98 | % some dot-assignments are contained in the defaults 99 | try 100 | res = struct(); 101 | for k=1:length(defnames) 102 | if any(defnames{k}=='.') 103 | eval(['res.' defnames{k} ' = defvalues{k};']); 104 | else 105 | res.(defnames{k}) = defvalues{k}; 106 | end 107 | end 108 | catch 109 | error(['invalid field name specified in defaults: ' defnames{k}]); 110 | end 111 | end 112 | else 113 | res = struct(); 114 | end 115 | 116 | % --- handle overrides --- 117 | if ~isempty(args) 118 | % splice substructs into the name-value list 119 | if any(cellfun('isclass',args(1:2:end),'struct')) 120 | args = flatten_structs(args); end 121 | 122 | % rewrite alternative names into their standard form... 123 | if exist('name_for_alternative','var') 124 | for k=1:2:length(args) 125 | if isfield(name_for_alternative,args{k}) 126 | args{k} = name_for_alternative.(args{k}); end 127 | end 128 | end 129 | 130 | % override defaults with arguments... 131 | try 132 | if [args{1:2:end}]~='.' 133 | for k=1:2:length(args) 134 | res.(args{k}) = args{k+1}; end 135 | else 136 | % some dot-assignments are contained in the overrides 137 | for k=1:2:length(args) 138 | if any(args{k}=='.') 139 | eval(['res.' args{k} ' = args{k+1};']); 140 | else 141 | res.(args{k}) = args{k+1}; 142 | end 143 | end 144 | end 145 | catch 146 | if ischar(args{k}) 147 | error(['invalid field name specified in arguments: ' args{k}]); 148 | else 149 | error(['invalid field name specified for the argument at position ' num2str(k)]); 150 | end 151 | end 152 | end 153 | 154 | % check for missing but mandatory args 155 | % note: the used string needs to match mandatory.m 156 | missing_entries = strcmp('__arg_mandatory__',struct2cell(res)); 157 | if any(missing_entries) 158 | fn = fieldnames(res)'; 159 | fn = fn(missing_entries); 160 | error(['The parameters {' sprintf('%s, ',fn{1:end-1}) fn{end} '} were unspecified but are mandatory.']); 161 | end 162 | 163 | % copy to the caller's workspace if no output requested 164 | if nargout == 0 165 | for fn=fieldnames(res)' 166 | assignin('caller',fn{1},res.(fn{1})); end 167 | end 168 | 169 | 170 | % substitute any structs in place of a name-value pair into the name-value list 171 | function args = flatten_structs(args) 172 | k = 1; 173 | while k <= length(args) 174 | if isstruct(args{k}) 175 | tmp = [fieldnames(args{k}) struct2cell(args{k})]'; 176 | args = [args(1:k-1) tmp(:)' args(k+1:end)]; 177 | k = k+numel(tmp); 178 | else 179 | k = k+2; 180 | end 181 | end 182 | -------------------------------------------------------------------------------- /private/sphericalSplineInterpolate.m: -------------------------------------------------------------------------------- 1 | function [W,Gss,Gds,Hds]=sphericalSplineInterpolate(src,dest,lambda,order,type,tol) 2 | %interpolate matrix for spherical interpolation 3 | % 4 | % W = sphericalSplineInterpolate(src,dest,lambda,order,type,tol) 5 | % 6 | % Inputs: 7 | % src - [3 x N] old electrode positions 8 | % dest - [3 x M] new electrode positions 9 | % lambda - [float] regularisation parameter for smoothing the estimates (1e-5) 10 | % order - [float] order of the polynomial interplotation to use (4) 11 | % type - [str] one of; ('spline') 12 | % 'spline' - spherical Spline 13 | % 'slap' - surface Laplician (aka. CSD) 14 | % tol - [float] tolerance for the legendre poly approx (1e-7) 15 | % Outputs: 16 | % W - [M x N] linear mapping matrix between old and new co-ords 17 | % 18 | % Based upon the paper: Perrin89 19 | 20 | % Copyright 2009- by Jason D.R. Farquhar (jdrf@zepler.org) 21 | % Permission is granted for anyone to copy, use, or modify this 22 | % software and accompanying documents, provided this copyright 23 | % notice is retained, and note is made of any changes that have been 24 | % made. This software and documents are distributed without any 25 | % warranty, express or implied. 26 | 27 | if ( nargin < 3 || isempty(lambda) ) lambda=1e-5; end; 28 | if ( nargin < 4 || isempty(order) ) order=4; end; 29 | if ( nargin < 5 || isempty(type)) type='spline'; end; 30 | if ( nargin < 6 || isempty(tol) ) tol=eps; end; 31 | 32 | % map the positions onto the sphere (not using repop, by JMH) 33 | src = src ./repmat(sqrt(sum(src.^2)), size(src, 1), 1); % src = repop(src,'./',sqrt(sum(src.^2))); 34 | dest = dest./repmat(sqrt(sum(dest.^2)), size(dest, 1), 1); % dest = repop(dest,'./',sqrt(sum(dest.^2))); 35 | 36 | %calculate the cosine of the angle between the new and old electrodes. If 37 | %the vectors are on top of each other, the result is 1, if they are 38 | %pointing the other way, the result is -1 39 | cosSS = src'*src; % angles between source positions 40 | cosDS = dest'*src; % angles between destination positions 41 | 42 | % Compute the interpolation matrix to tolerance tol 43 | [Gss] = interpMx(cosSS,order,tol); % [nSrc x nSrc] 44 | [Gds Hds] = interpMx(cosDS,order,tol); % [nDest x nSrc] 45 | 46 | % Include the regularisation 47 | if ( lambda>0 ) Gss = Gss+lambda*eye(size(Gss)); end; 48 | 49 | % Compute the mapping to the polynomial coefficients space % [nSrc+1 x nSrc+1] 50 | % N.B. this can be numerically unstable so use the PINV to solve.. 51 | muGss=1;%median(diag(Gss)); % used to improve condition number when inverting. Probably uncessary 52 | %C = [ Gss muGss*ones(size(Gss,1),1)]; 53 | C = [ Gss muGss*ones(size(Gss,1),1);... 54 | muGss*ones(1,size(Gss,2)) 0]; 55 | iC = pinv(C); 56 | 57 | % Compute the mapping from source measurements and positions to destination positions 58 | if ( strcmp(lower(type),'spline') ) 59 | W = [Gds ones(size(Gds,1),1).*muGss]*iC(:,1:end-1); % [nDest x nSrc] 60 | elseif (strcmp(lower(type),'slap')) 61 | W = Hds*iC(1:end-1,1:end-1);%(:,1:end-1); % [nDest x nSrc] 62 | end 63 | return; 64 | %-------------------------------------------------------------------------- 65 | function [G,H]=interpMx(cosEE,order,tol) 66 | % compute the interpolation matrix for this set of point pairs 67 | if ( nargin < 3 || isempty(tol) ) tol=1e-10; end; 68 | G=zeros(size(cosEE)); H=zeros(size(cosEE)); 69 | for i=1:numel(cosEE); 70 | x = cosEE(i); 71 | n=1; Pns1=1; Pn=x; % seeds for the legendre ploy recurence 72 | tmp = ( (2*n+1) * Pn ) / ((n*n+n).^order); 73 | G(i) = tmp ; % 1st element in the sum 74 | H(i) = (n*n+n)*tmp; % 1st element in the sum 75 | oGi=inf; dG=abs(G(i)); oHi=inf; dH=abs(H(i)); 76 | for n=2:500; % do the sum 77 | Pns2=Pns1; Pns1=Pn; Pn=((2*n-1)*x*Pns1 - (n-1)*Pns2)./n; % legendre poly recurance 78 | oGi=G(i); oHi=H(i); 79 | tmp = ((2*n+1) * Pn) / ((n*n+n).^order) ; 80 | G(i) = G(i) + tmp; % update function estimate, spline interp 81 | H(i) = H(i) + (n*n+n)*tmp; % update function estimate, SLAP 82 | dG = (abs(oGi-G(i))+dG)/2; dH=(abs(oHi-H(i))+dH)/2; % moving ave gradient est for convergence 83 | %fprintf('%d) dG =%g \t dH = %g\n',n,dG,dH);%abs(oGi-G(i)),abs(oHi-H(i))); 84 | if ( dG 62 | X = U*V'; 63 | [U0, S0, ~] = eigs(L + alpha*diag(L\(sum(X.^2, 2))), k); %,'sm'); %#ok 64 | X0 = U0; 65 | 66 | % Call manoptsolve to automatically call an appropriate solver. 67 | % Note: it calls the trust regions solver as we have all the required 68 | % ingredients, namely, gradient and Hessian, information. 69 | Xsol = manoptsolve(problem, X0); 70 | end 71 | -------------------------------------------------------------------------------- /readme-asr.txt: -------------------------------------------------------------------------------- 1 | This is a reference implementation of the Artifact Subspace Reconstruction (ASR) method in MATLAB. 2 | 3 | For convenience a few wrapper functions that call the ASR core functions are included in 4 | the sub-directory called "extras". These are not part of the ASR method but are useful to get started or 5 | can serve as code samples. See also readme-extras.txt for basic usage examples of those functions. 6 | 7 | 8 | Using the core implementation directly 9 | ====================================== 10 | 11 | The reference implementation may require the Signal Processing toolbox or alternatively a pre-computed 12 | IIR filter kernel if spectrally weighted statistics should be used (this is an optional feature; the filter 13 | kernel can be set to A=1 and B=1 to disable spectral weighting). 14 | 15 | In all cases the signal that is passed to the method should be zero-mean, i.e., first high-pass filtered if necessary. 16 | In the following code samples the data is randomly generated with zero mean. 17 | 18 | The general calling convention of the method is: 19 | 20 | 1) Calibrate the parameters using the asr_calibrate function and some reasonably clean 21 | calibration data, e.g., resting-state EEG, as in the following code snippet. 22 | The recommended data length is ca. 1 minute or longer, the absolute minimum would be ca. 15 seconds. 23 | There are optional parameters as documented in the function, in particular a tunable threshold parameter 24 | governing the aggressiveness of the cleaning (although the defaults are sensible for testing purposes). 25 | 26 | calibdata = randn(20,10000); % simulating 20-channel, 100-second random data at 100 Hz 27 | state = asr_calibrate(calibdata,100) % calibrate the parameters of the method (sampling rate 100Hz) 28 | 29 | 30 | 2a) Apply the processing function to data that shall be cleaned, 31 | either in one large block (entire recording) as follows... 32 | 33 | rawdata = randn(20,1000000); % simulating random data to process 34 | cleandata = asr_process(rawdata,100,state); % apply the processing to the data (sampling rate 100Hz); see documentation of the function for optional parameters. 35 | 36 | Also note that for offline processing the method may be calibrated on the same data that should also be 37 | cleaned, as long as the fraction of contaminated data is lower than a theoretical upper bound of 38 | 50% (a conservative empirical estimate would be approx. 30%). For extremely contaminated data one may 39 | extract one or more clean segments from the data and calibrate on those. 40 | 41 | 42 | 2b) ... or alternatively apply the processing function to data in an online/incremental (chunk-by-chunk) 43 | fashion, as follows: 44 | 45 | while 1 46 | newchunk = randn(20,50); % here using 0.5-second chunks; the chunk length is near-arbitrary and may vary, but transitions will be sharper for very short chunks. 47 | [cleanchunk,state] = asr_process(newchunk,100,state); % apply the processing to the data and update the filter state (as in MATLAB's filter()) 48 | end 49 | 50 | 51 | -------------------------------------------------------------------------------- /readme-extras.txt: -------------------------------------------------------------------------------- 1 | Overview 2 | ======== 3 | 4 | In the following, several more extensive data analysis pathways involving EEGLAB functions are sketched. The first pathway 5 | illustrates the use of ASR through a convenience wrapper function (clean_asr) that simplifies some of the logistics 6 | of running the core functions. The second pathway shows the use of an all-in-one cleaning function that, besides ASR 7 | also performs some basic pre- and post-processing such as removing bad channels and time windows (clean_artifacts). 8 | The last example invokes a function that can be used to visually compare raw versus cleaned data (vis_artifacts). 9 | 10 | All of these examples accept data in the form of EEGLAB dataset structs (see http://sccn.ucsd.edu/wiki/A05:_Data_Structures). 11 | The easiest way to supply the necessary data is to use the EEGLAB toolbox (freely available from http://sccn.ucsd.edu/eeglab/). 12 | The data set used in these examples is the freely available Sternberg data set #1. This data set can be downloaded 13 | from headit.org by navigating to the study named "Modified Sternberg Working Memory Task", going to subject #1, 14 | and then downloading the first recording of the subject, here called "eeg_recording_1.SMA" -- the following link 15 | points directly to this data set but may be deprecated by the database maintainers at some point: 16 | http://headit-beta.ucsd.edu/attachments/e9e23582-a236-11e2-9420-0050563f2612/download). 17 | 18 | These examples use additional code (in the extras directory) developed at the Swartz Center for which we cannot provide the 19 | same level of support as for the core ASR functions. 20 | 21 | Using ASR through the convenience wrapper 22 | ========================================= 23 | 24 | addpath(pwd); % add current directory to the path 25 | addpath extras; % add extras to the path 26 | eeglab; % start EEGLAB if not done so already (freely available from sccn.ucsd.edu/eeglab/) 27 | 28 | raw = pop_biosig('eeg_recording_1.SMA'); % load a sample data set (freely available from: headit.org or http://headit-beta.ucsd.edu/recording_sessions/e67fc0e4-a236-11e2-9420-0050563f2612) 29 | highpassed = clean_drifts(raw,[0.25 0.75]); % high-pass filter it using a 0.25-0.75 Hz transition band 30 | cleaned = clean_asr(highpassed,2); % run the ASR algorithm using a threshold of 2 standard deviations (somewhat less conservative than default; see docs for more options) 31 | 32 | 33 | Using ASR as part of the all-in-one-cleaning 34 | ============================================ 35 | 36 | addpath(pwd); % add current directory to the path 37 | addpath extras; % add extras to the path 38 | eeglab; % start EEGLAB if not done so already (freely available from sccn.ucsd.edu/eeglab/) 39 | 40 | raw = pop_biosig('eeg_recording_1.SMA'); % load a sample data set (freely available from: headit.org or http://headit-beta.ucsd.edu/recording_sessions/e67fc0e4-a236-11e2-9420-0050563f2612) 41 | cleaned = clean_artifacts(raw,'BurstCriterion',2,'WindowCriterion','off'); % clean the data with final removal of windows disabled and same threshold as before 42 | 43 | 44 | Visually comparing the raw and cleaned data 45 | =========================================== 46 | 47 | vis_artifacts(cleaned,raw); % this function has important keyboard shortcuts to toggle the display -- see documentation 48 | 49 | -------------------------------------------------------------------------------- /vis_artifacts.m: -------------------------------------------------------------------------------- 1 | function [h_old,h_new] = vis_artifacts(new,old,varargin) 2 | % vis_artifacts(NewEEG,OldEEG,Options...) 3 | % Display the artifact rejections done by any of the artifact cleaning functions. 4 | % 5 | % Keyboard Shortcuts: 6 | % [n] : display just the new time series 7 | % [o] : display just the old time series 8 | % [b] : display both time series super-imposed 9 | % [d] : display the difference between both time series 10 | % [+] : increase signal scale 11 | % [-] : decrease signal scale 12 | % [*] : expand time range 13 | % [/] : reduce time range 14 | % [h] : show/hide slider 15 | % [e] : toggle events 16 | % [l] : toggle event legend 17 | % 18 | % In: 19 | % NewEEG : cleaned continuous EEG data set 20 | % OldEEG : original continuous EEG data set 21 | % Options... : name-value pairs specifying the options, with names: 22 | % 'YRange' : y range of the figure that is occupied by the signal plot 23 | % 'YScaling' : distance of the channel time series from each other in std. deviations 24 | % 'WindowLength : window length to display 25 | % 'NewColor' : color of the new (i.e., cleaned) data 26 | % 'OldColor' : color of the old (i.e., uncleaned) data 27 | % 'HighpassOldData' : whether to high-pass the old data if not already done 28 | % 'ScaleBy' : the data set according to which the display should be scaled, can be 29 | % 'old', 'new' or 'noscale' (default: 'new') 30 | % 'ChannelSubset' : optionally a channel subset to display 31 | % 'TimeSubet' : optionally a time subrange to display 32 | % 'DisplayMode' : what should be displayed: 'both', 'new', 'old', 'diff' 33 | % 'ShowSetname' : whether to display the dataset name in the title 34 | % 'EqualizeChannelScaling' : optionally equalize the channel scaling 35 | % See also code for more options. 36 | % 37 | % Notes: 38 | % This function is primarily meant for testing purposes and is not a particularly high-quality 39 | % implementation. If you are a MATLAB expert and have an interest in this function we would 40 | % appreciate it if you chose to do a clean rewrite. :) 41 | % 42 | % Examples: 43 | % vis_artifacts(clean,raw) 44 | % 45 | % % display only a subset of channels 46 | % vis_artifacts(clean,raw,'ChannelSubset',1:4:raw.nbchan); 47 | % 48 | % 49 | % Christian Kothe, Swartz Center for Computational Neuroscience, UCSD 50 | % 2010-07-06 51 | % 52 | % relies on the findjobj() function by Yair M. Altman. 53 | % 54 | % 07/27/2018 Makoto. Addressed the issue of ResizeFcn behavior change started in R2014b. https://www.mathworks.com/help/matlab/graphics_transition/why-has-the-behavior-of-resizefcn-changed.html 55 | 56 | % Copyright (C) Christian Kothe, SCCN, 2012, ckothe@ucsd.edu 57 | % 58 | % This program is free software; you can redistribute it and/or modify it under the terms of the GNU 59 | % General Public License as published by the Free Software Foundation; either version 2 of the 60 | % License, or (at your option) any later version. 61 | % 62 | % This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without 63 | % even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 64 | % General Public License for more details. 65 | % 66 | % You should have received a copy of the GNU General Public License along with this program; if not, 67 | % write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 68 | % USA 69 | 70 | have_signallegend = false; 71 | have_eventlegend = false; 72 | 73 | if nargin < 2 74 | old = new; 75 | elseif ischar(old) 76 | varargin = [{old} varargin]; 77 | old = new; 78 | end 79 | 80 | % parse options 81 | opts = hlp_varargin2struct(varargin, ... 82 | {'yrange','YRange'}, [0.05 0.95], ... % y range of the figure occupied by the signal plot 83 | {'yscaling','YScaling'}, 3.5, ... % distance of the channel time series from each other in std. deviations 84 | {'wndlen','WindowLength'}, 10, ... % window length to display 85 | {'newcol','NewColor'}, [0 0 0.5], ... % color of the new (i.e., cleaned) data 86 | {'oldcol','OldColor'}, [1 0 0], ... % color of the old (i.e., uncleaned) data 87 | {'highpass_old','HighpassOldData'},true, ...% whether to high-pass the old data if not already done 88 | {'show_removed_portions','ShowRemovedPortions'},true, ...% whether to show removed data portions (if only one set is passed in) 89 | {'show_events','ShowEvents'},true, ... % whether to show events 90 | {'show_eventlegend','ShowEventLegend'},false, ... % whether to show a legend for the currently visible events 91 | {'scale_by','ScaleBy'},'allnew',... % the data set according to which the display should be scaled (can be allold, allnew, wndold, wndnew or noscale) 92 | {'channel_subset','ChannelSubset'},[], ... % optionally a channel subset to display 93 | {'time_subset','TimeSubset'},[],... % optionally a time subrange to display 94 | {'display_mode','DisplayMode'},'both',... % what should be displayed: 'both', 'new', 'old', 'diff' 95 | {'show_setname','ShowSetname'},true,... % whether to display the dataset name in the title 96 | {'line_spec','LineSpec'},'-',... % line style for plotting 97 | {'line_width','LineWidth'},0.5,... % line width 98 | {'add_legend','AddLegend'},false,... % add a signal legend 99 | {'equalize_channel_scaling','EqualizeChannelScaling'},false); % optionally equalize the channel scaling 100 | 101 | % ensure that the data are not epoched and expand the rejections with NaN's (now both should have the same size) 102 | if opts.show_removed_portions 103 | new = expand_rejections(to_continuous(new)); 104 | old = expand_rejections(to_continuous(old)); 105 | end 106 | new.chanlocs = old.chanlocs; 107 | 108 | % correct for filter delay 109 | if isfield(new.etc,'filter_delay') 110 | new.data = new.data(:,[1+round(new.etc.filter_delay*new.srate):end end:-1:(end+1-round(new.etc.filter_delay*new.srate))]); end 111 | if isfield(old.etc,'filter_delay') 112 | old.data = old.data(:,[1+round(old.etc.filter_delay*old.srate):end end:-1:(end+1-round(old.etc.filter_delay*old.srate))]); end 113 | 114 | % make sure that the old data is high-passed the same way as the new data 115 | if opts.highpass_old && isfield(new.etc,'clean_drifts_kernel') && ~isfield(old.etc,'clean_drifts_kernel') 116 | old.data = old.data'; 117 | for c=1:old.nbchan 118 | old.data(:,c) = filtfilt_fast(new.etc.clean_drifts_kernel,1,old.data(:,c)); end 119 | old.data = old.data'; 120 | end 121 | 122 | if isscalar(opts.line_width) 123 | opts.line_width = [opts.line_width opts.line_width]; end 124 | 125 | % optionally pick a subrange to work on 126 | if ~isempty(opts.channel_subset) 127 | old = pop_select(old,'channel',opts.channel_subset); 128 | new = pop_select(new,'channel',opts.channel_subset); 129 | end 130 | 131 | if ~isempty(opts.time_subset) 132 | old = pop_select(old,'time',opts.time_subset); 133 | new = pop_select(new,'time',opts.time_subset); 134 | end 135 | 136 | if opts.equalize_channel_scaling 137 | rescale = 1./mad(old.data,[],2); 138 | new.data = bsxfun(@times,new.data,rescale); 139 | old.data = bsxfun(@times,old.data,rescale); 140 | end 141 | 142 | % generate event colormap - fixed for numeric event type by Makoto 5/15/14 143 | if ~isempty(old.event) 144 | if isnumeric(old.event(1).type) 145 | for tmpIdx = 1:length(old.event) 146 | old.event(tmpIdx).type = num2str(old.event(tmpIdx).type); 147 | end 148 | end 149 | opts.event_colormap = gen_colormap(old.event,'jet'); 150 | end 151 | 152 | % calculate whole-data scale 153 | old_iqr = 2*mad(quantile(old.data',1000),1)'; 154 | old_iqr(isnan(old_iqr)) = deal(mean(old_iqr(~isnan(old_iqr)))); 155 | new_iqr = 2*mad(quantile(new.data',1000),1)'; 156 | new_iqr(isnan(new_iqr)) = deal(mean(new_iqr(~isnan(new_iqr)))); 157 | 158 | % create figure & slider 159 | lastPos = 0; 160 | %hFig = figure('ResizeFcn',@on_window_resized,'KeyPressFcn',@(varargin)on_key(varargin{2}.Key)); hold; axis(); 161 | hFig = figure('KeyPressFcn',@(varargin)on_key(varargin{2}.Key)); hold; axis(); 162 | hAxis = gca; 163 | hSlider = uicontrol('style','slider','KeyPressFcn',@(varargin)on_key(varargin{2}.Key)); on_resize(); 164 | set(hSlider, 'callback', @on_update); 165 | % jSlider = findjobj(hSlider); 166 | % jSlider.AdjustmentValueChangedCallback = @on_update; 167 | 168 | annotation(hFig,'textbox', [0 0.07 1 0],... 169 | 'String', {'Keyboard shortcuts: [n] new data, [o] old data, [b] both data, [d] difference, [+] increase amp scale, [-] decrease amp scale, [*] shrink time scale, [/] expand time scale, [h] show/hide slider.'},... 170 | 'HorizontalAlignment','center', 'FontSize',14, 'FitBoxToText','off', 'LineStyle','none'); 171 | 172 | % Implementing a delay fuse (07/27/2017 Makoto). 173 | set(hFig, 'ResizeFcn', @on_window_resized); 174 | 175 | % do the initial update 176 | %on_update(); 177 | 178 | 179 | function repaint(relPos,moved) 180 | % repaint the current data 181 | 182 | % if this happens, we are maxing out MATLAB's graphics pipeline: let it catch up 183 | if relPos == lastPos && moved 184 | return; end 185 | 186 | % axes 187 | cla(hAxis); 188 | gca = hAxis; 189 | 190 | % compute pixel range from axis properties 191 | xl = get(hAxis,'XLim'); 192 | yl = get(hAxis,'YLim'); 193 | fp = get(hFig,'Position'); 194 | ap = get(hAxis,'Position'); 195 | pixels = (fp(3))*(ap(3)-ap(1)); 196 | ylr = yl(1) + opts.yrange*(yl(2)-yl(1)); 197 | channel_y = (ylr(2):(ylr(1)-ylr(2))/(size(new.data,1)-1):ylr(1))'; 198 | 199 | % Add channel labels to y axis 200 | if isfield(old.chanlocs,'labels') 201 | set(hAxis,'ytick',flipud(channel_y)); 202 | set(hAxis,'yticklabel',fliplr({old.chanlocs.labels})); 203 | end 204 | 205 | % compute sample range 206 | wndsamples = opts.wndlen * new.srate; 207 | pos = floor((size(new.data,2)-wndsamples)*relPos); 208 | wndindices = 1 + floor(0:wndsamples/pixels:(wndsamples-1)); 209 | wndrange = pos+wndindices; 210 | wndrange(wndrange > length(old.data)) = length(old.data); 211 | 212 | oldwnd = old.data(:,wndrange); 213 | newwnd = new.data(:,wndrange); 214 | switch opts.scale_by 215 | case 'allnew' 216 | iqrange = new_iqr; 217 | case 'allold' 218 | iqrange = old_iqr; 219 | case {'wndnew','new'} 220 | iqrange = mad(newwnd',1)'; 221 | iqrange(isnan(iqrange)) = mad(oldwnd(isnan(iqrange),:)',1)'; 222 | case {'wndold','old'} 223 | iqrange = mad(oldwnd',1)'; 224 | case 'noscale' 225 | iqrange = ones(size(new.data,1),1); 226 | otherwise 227 | error('Unsupported scale_by option.'); 228 | end 229 | scale = ((ylr(2)-ylr(1))/size(new.data,1)) ./ (opts.yscaling*iqrange); scale(~isfinite(scale)) = 0; 230 | scale(scale>median(scale)*3) = median(scale); 231 | scale = scale(:); 232 | scale = repmat(scale,1,length(wndindices)); 233 | 234 | % draw 235 | if opts.show_setname 236 | tit = sprintf('%s - ',[old.filepath filesep old.filename]); 237 | else 238 | tit = ''; 239 | end 240 | 241 | if ~isempty(wndrange) 242 | tit = [tit sprintf('[%.1f - %.1f]',new.xmin + (wndrange(1)-1)/new.srate, new.xmin + (wndrange(end)-1)/new.srate)]; 243 | end 244 | 245 | xrange = xl(1):(xl(2)-xl(1))/(length(wndindices)-1):xl(2); 246 | yoffset = repmat(channel_y,1,length(wndindices)); 247 | switch opts.display_mode 248 | case 'both' 249 | title([tit '; superposition'],'Interpreter','none'); 250 | h_old = plot(xrange, (yoffset + scale.*oldwnd)','Color',opts.oldcol,'LineWidth',opts.line_width(1)); 251 | h_new = plot(xrange, (yoffset + scale.*newwnd)','Color',opts.newcol,'LineWidth',opts.line_width(2)); 252 | case 'new' 253 | title([tit '; cleaned'],'Interpreter','none'); 254 | plot(xrange, (yoffset + scale.*newwnd)','Color',opts.newcol,'LineWidth',opts.line_width(2)); 255 | case 'old' 256 | title([tit '; original'],'Interpreter','none'); 257 | plot(xrange, (yoffset + scale.*oldwnd)','Color',opts.oldcol,'LineWidth',opts.line_width(1)); 258 | case 'diff' 259 | title([tit '; difference'],'Interpreter','none'); 260 | plot(xrange, (yoffset + scale.*(oldwnd-newwnd))','Color',opts.newcol,'LineWidth',opts.line_width(1)); 261 | end 262 | 263 | % also plot events 264 | if opts.show_events && ~isempty(old.event) 265 | evtlats = [old.event.latency]; 266 | evtindices = find(evtlats>wndrange(1) & evtlats