├── .gitignore ├── load_xdf_innerloop.mexw64 ├── load_xdf_innerloop.mexmaca64 ├── load_xdf_innerloop.mexmaci64 ├── LICENSE ├── readme.md ├── load_xdf_innerloop.c └── load_xdf.m /.gitignore: -------------------------------------------------------------------------------- 1 | .DS_Store 2 | xdfimport*.zip 3 | xdf.zip 4 | .idea 5 | *.asv 6 | *.m~ 7 | -------------------------------------------------------------------------------- /load_xdf_innerloop.mexw64: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xdf-modules/xdf-Matlab/HEAD/load_xdf_innerloop.mexw64 -------------------------------------------------------------------------------- /load_xdf_innerloop.mexmaca64: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xdf-modules/xdf-Matlab/HEAD/load_xdf_innerloop.mexmaca64 -------------------------------------------------------------------------------- /load_xdf_innerloop.mexmaci64: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xdf-modules/xdf-Matlab/HEAD/load_xdf_innerloop.mexmaci64 -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | BSD 2-Clause License 2 | 3 | Copyright (c) 2015, Alejandro Ojeda and Christian Kothe 4 | All rights reserved. 5 | 6 | Redistribution and use in source and binary forms, with or without 7 | modification, are permitted provided that the following conditions are met: 8 | 9 | * Redistributions of source code must retain the above copyright notice, this 10 | list of conditions and the following disclaimer. 11 | 12 | * Redistributions in binary form must reproduce the above copyright notice, 13 | this list of conditions and the following disclaimer in the documentation 14 | and/or other materials provided with the distribution. 15 | 16 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 17 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 18 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 19 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 20 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 21 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 22 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 23 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 24 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 25 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 26 | -------------------------------------------------------------------------------- /readme.md: -------------------------------------------------------------------------------- 1 | # Overview 2 | 3 | This is a MATLAB importer for .xdf files. xdf files are likely to have been created by [LabRecorder](https://github.com/labstreaminglayer/App-LabRecorder), the default file recorder for use with [LabStreamingLayer](https://github.com/sccn/labstreaminglayer). LabRecorder records a collection of streams, including all their data (time-series / markers, meta-data) into a single XDF file. The XDF format (Extensible Data Format) is a general-purpose format for time series and their meta-data that was jointly developed with LSL to ensure compatibility, see [here](http://github.com/sccn/xdf/). 4 | 5 | # Usage 6 | 7 | After a session has been recorded to disk using the LabRecorder or any other compatible recording application, it can be imported into MATLAB using the functions in this folder. 8 | 9 | Note that EEGLAB plugins are structured so the EEGLAB (/BCILAB/MoBILAB) plugin files are in the top level (i.e. the directory containing this readme) and the actual import function is `load_xdf` in the `xdf` subfolder. 10 | 11 | To use `load_xdf` directly: 12 | 13 | * Start MATLAB and make sure that you have the function added to MATLAB's path. You can do this either through the GUI, under File / Set Path / Add Folder) or in the command line, by typing: 14 | 15 | > `addpath('C:\\path\\to\\xdf_repo\\Matlab\\xdf')` 16 | 17 | * To load an .xdf file, type in the command line: 18 | 19 | > `streams = load_xdf('your_file_name.xdf')` 20 | 21 | * After a few seconds it should return a cell array with one cell for every stream that was contained in the file. For each stream you get a struct that contains the entire meta-data (including channel descriptions and domain-specific information), as well as the time series data itself (numeric or cell-string array, depending on the value type of the stream), and the time stamps of each sample in the time series. All time stamps (across all streams, even if they were collected on different computers of the lab network) are in the same time domain, so they are synchronized. Note that time stamps from different .xdf files are generally not synchronized (although they will normally be in seconds since the recording machine was turned on). 22 | 23 | # Documentation 24 | As usual, in MATLAB, to get the documentation of the function, type `help load_xdf` or `doc load_xdf`. 25 | -------------------------------------------------------------------------------- /load_xdf_innerloop.c: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | /* 5 | * [Values,Timestamps] = load_xdf_innerloop(Data, NumChannels, FormatString, SamplingInterval, LastTimestamp); 6 | * MEX kernel that implements the inner loop of the load_xdf function. 7 | * 8 | * In: 9 | * Data : byte array that contains the data of the chunk 10 | * 11 | * NumChannels : number of channels per sample (double) 12 | * 13 | * FormatString : format string of the data (determines the format of the data values); can be one of the following: 14 | * - '*int8' 15 | * - '*int16' 16 | * - '*int32' 17 | * - '*int64' 18 | * - '*float32' 19 | * - '*double64' 20 | * - '*string' 21 | * 22 | * SamplingInterval : sampling interval in seconds (can be 0, if irregular) 23 | * 24 | * LastTimestamp : time stamp of the last sample preceding the data, in seconds 25 | * 26 | * Out: 27 | * Values : [#Channels x #Samples] array containing the chunk data; either numeric or cell array 28 | * 29 | * Timestamps: #Samples array of the time stamps for every sample, in seconds 30 | * 31 | * Christian Kothe, Swartz Center for Computational Neuroscience, UCSD 32 | * 2012-08-06 33 | */ 34 | 35 | /* Value format of a channel. */ 36 | typedef enum { 37 | cft_undefined = 0, 38 | cft_float32 = 1, 39 | cft_double64 = 2, 40 | cft_string = 3, 41 | cft_int32 = 4, 42 | cft_int16 = 5, 43 | cft_int8 = 6, 44 | cft_int64 = 7, 45 | } value_format_t; 46 | 47 | /* Number of bytes occupied by a single value (excluding strings, which are variable-length). */ 48 | unsigned value_bytes[] = {0,4,8,0,4,2,1,8}; 49 | 50 | void mexFunction( int nlhs, mxArray *plhs[], 51 | int nrhs, const mxArray*prhs[] ) 52 | { 53 | /* input arguments */ 54 | unsigned char *data; 55 | unsigned num_channels; 56 | char format_string[1024]; 57 | double sampling_interval; 58 | double last_timestamp; 59 | 60 | /* output arguments */ 61 | unsigned char *values; 62 | double *timestamps; 63 | 64 | /* variables */ 65 | mwSize num_samples, s, c, sample_bytes; 66 | value_format_t format_code; 67 | mwSize string_dims[2]; 68 | 69 | /* temporaries */ 70 | unsigned bytes, num_chars; 71 | char *tmp; 72 | 73 | /* check input/output argument formats */ 74 | if (nrhs != 5) 75 | mexErrMsgTxt("5 input arguments required."); 76 | if (nlhs != 2) 77 | mexErrMsgTxt("2 output arguments required."); 78 | 79 | /* get inputs */ 80 | data = (unsigned char*)mxGetData(prhs[0]); 81 | num_channels = (unsigned)(*((double*)mxGetData(prhs[1]))); 82 | mxGetNChars(prhs[2], format_string, mxGetNumberOfElements(prhs[2])+1); 83 | sampling_interval = *((double*)mxGetData(prhs[3])); 84 | last_timestamp = *((double*)mxGetData(prhs[4])); 85 | 86 | /* set the format code */ 87 | format_code = cft_undefined; 88 | if (strcmp(format_string,"*float32") == 0) 89 | format_code = cft_float32; 90 | if (strcmp(format_string,"*double64") == 0) 91 | format_code = cft_double64; 92 | if (strcmp(format_string,"*string") == 0) 93 | format_code = cft_string; 94 | if (strcmp(format_string,"*int32") == 0) 95 | format_code = cft_int32; 96 | if (strcmp(format_string,"*int16") == 0) 97 | format_code = cft_int16; 98 | if (strcmp(format_string,"*int8") == 0) 99 | format_code = cft_int8; 100 | if (strcmp(format_string,"*int64") == 0) 101 | format_code = cft_int64; 102 | if (format_code == cft_undefined) 103 | mexErrMsgTxt("The FormatString does not contain a recognized format."); 104 | 105 | /* read number of samples (as a varlen int) */ 106 | bytes = *data++; 107 | if (bytes == 1) 108 | num_samples = *data++; 109 | if (bytes == 4) { 110 | num_samples = *(uint32_t*)data; 111 | data += 4; 112 | } 113 | if (bytes == 8) 114 | mexErrMsgTxt("This importer cannot yet handle chunks with more than 4 billion samples."); 115 | 116 | /* allocate memory for the time stamps*/ 117 | plhs[1] = mxCreateNumericMatrix(1,num_samples,mxDOUBLE_CLASS,mxREAL); 118 | timestamps = (double*)mxGetData(plhs[1]); 119 | if (format_code != cft_string) { 120 | /* numeric data case: allocate output array */ 121 | switch(format_code) { 122 | case cft_float32: 123 | plhs[0] = mxCreateNumericMatrix(num_channels,num_samples,mxSINGLE_CLASS,mxREAL); 124 | break; 125 | case cft_double64: 126 | plhs[0] = mxCreateNumericMatrix(num_channels,num_samples,mxDOUBLE_CLASS,mxREAL); 127 | break; 128 | case cft_int32: 129 | plhs[0] = mxCreateNumericMatrix(num_channels,num_samples,mxINT32_CLASS,mxREAL); 130 | break; 131 | case cft_int16: 132 | plhs[0] = mxCreateNumericMatrix(num_channels,num_samples,mxINT16_CLASS,mxREAL); 133 | break; 134 | case cft_int8: 135 | plhs[0] = mxCreateNumericMatrix(num_channels,num_samples,mxINT8_CLASS,mxREAL); 136 | break; 137 | case cft_int64: 138 | plhs[0] = mxCreateNumericMatrix(num_channels,num_samples,mxINT64_CLASS,mxREAL); 139 | break; 140 | default: 141 | mexErrMsgTxt("Unrecognized data type."); 142 | } 143 | sample_bytes = value_bytes[format_code]*num_channels; 144 | values = (unsigned char*)mxGetData(plhs[0]); 145 | /* for each sample... */ 146 | for (s=0;s 365 | % append to the time series... 366 | temp(id).time_series{end+1} = values; 367 | temp(id).time_stamps{end+1} = timestamps; 368 | catch e 369 | % an error occurred (perhaps a chopped-off file): emit a warning 370 | % and scan forward to the next recognized chunk. 371 | fprintf(' got error "%s" (%s), scanning forward to next boundary chunk.\n',e.identifier,e.message); 372 | scan_forward(f); 373 | end 374 | case 2 % read [StreamHeader] chunk 375 | % read [StreamId] 376 | streamid = fread(f,1,'uint32'); 377 | id = length(streams)+1; 378 | idmap(streamid) = id; 379 | % read [Content] 380 | data_uint = fread(f,len-6,'*uint8'); 381 | data = native2unicode(data_uint, 'UTF-8'); 382 | header = parse_xml_struct(data); 383 | 384 | if ~isfield(header.info, 'desc') 385 | header.info.desc = []; 386 | end 387 | streams{id} = header; 388 | if opts.Verbose 389 | fprintf([' found stream ' header.info.name '\n']); end 390 | % generate a few temporary fields 391 | temp(id).chns = str2num(header.info.channel_count); %#ok<*ST2NM> 392 | temp(id).srate = str2num(header.info.nominal_srate); 393 | temp(id).last_timestamp = 0; 394 | temp(id).time_series = {}; 395 | temp(id).time_stamps = {}; 396 | temp(id).clock_times = []; 397 | temp(id).offset_values = []; 398 | if temp(id).srate > 0 399 | temp(id).sampling_interval = 1/temp(id).srate; 400 | else 401 | temp(id).sampling_interval = 0; 402 | end 403 | % fread parsing format for data values 404 | temp(id).readfmt = ['*' header.info.channel_format]; 405 | if strcmp(temp(id).readfmt,'*double64') && ~have_mex 406 | temp(id).readfmt = '*double'; end % for fread() 407 | case 6 % read [StreamFooter] chunk 408 | % read [StreamId] 409 | id = idmap(fread(f,1,'uint32')); 410 | % read [Content] 411 | try 412 | footer = parse_xml_struct(fread(f,len-6,'*char')'); 413 | streams{id} = hlp_superimposedata(footer,streams{id}); 414 | catch e 415 | fprintf(' got error "%s" (%s), ignoring truncated XML structure.\n',e.identifier,e.message); 416 | end 417 | case 1 % read [FileHeader] chunk 418 | fileheader = parse_xml_struct(fread(f,len-2,'*char')'); 419 | case 4 % read [ClockOffset] chunk 420 | try 421 | % read [StreamId] 422 | id = idmap(fread(f,1,'uint32')); 423 | % read [CollectionTime] 424 | temp(id).clock_times(end+1) = fread(f,1,'double'); 425 | % read [OffsetValue] 426 | temp(id).offset_values(end+1) = fread(f,1,'double'); 427 | catch e 428 | % an error occurred (perhaps a chopped-off file): emit a 429 | % warning and scan forward to the next recognized chunk 430 | fprintf(' got error "%s" (%s), scanning forward to next boundary chunk.\n',e.identifier,e.message); 431 | scan_forward(f); 432 | end 433 | case 5 % read [Boundary] chunk 434 | fread(f, len-2, '*uint8'); 435 | otherwise 436 | % skip other chunk types 437 | fread(f,len-2,'*uint8'); 438 | end 439 | end 440 | 441 | % concatenate the signal across chunks 442 | for k=1:length(temp) 443 | try 444 | temp(k).time_series = [temp(k).time_series{:}]; 445 | temp(k).time_stamps = [temp(k).time_stamps{:}]; 446 | catch e 447 | disp(['Could not concatenate time series for stream ' streams{k}.info.name '; skipping.']); 448 | disp(['Reason: ' e.message]); 449 | temp(k).time_series = []; 450 | temp(k).time_stamps = []; 451 | end 452 | end 453 | 454 | 455 | % =================================================================== 456 | % === perform (fault-tolerant) clock synchronization if requested === 457 | % =================================================================== 458 | 459 | if opts.HandleClockSynchronization 460 | if opts.Verbose 461 | disp(' performing clock synchronization...'); end 462 | for k=1:length(temp) 463 | if ~isempty(temp(k).time_stamps) 464 | try 465 | clock_times = temp(k).clock_times; 466 | offset_values = temp(k).offset_values; 467 | if isempty(clock_times) 468 | error('No clock offset values present.'); end 469 | catch 470 | disp(['No clock offsets were available for stream "' streams{k}.info.name '"']); 471 | continue; 472 | end 473 | 474 | % detect clock resets (e.g., computer restarts during recording) if requested 475 | % this is only for cases where "everything goes wrong" during recording 476 | % note that this is a fancy feature that is not needed for normal XDF compliance 477 | if opts.HandleClockResets 478 | % first detect potential breaks in the synchronization data; this is only necessary when the 479 | % importer should be able to deal with recordings where the computer that served a stream 480 | % was restarted or hot-swapped during an ongoing recording, or the clock was reset otherwise 481 | time_diff = diff(clock_times); 482 | value_diff = abs(diff(offset_values)); 483 | % points where a glitch in the timing of successive clock measurements happened 484 | time_glitch = (time_diff < 0 | (((time_diff - median(time_diff)) ./ median(abs(time_diff-median(time_diff)))) > opts.ClockResetThresholdStds & ... 485 | ((time_diff - median(time_diff)) > opts.ClockResetThresholdSeconds))); 486 | % points where a glitch in successive clock value estimates happened 487 | value_glitch = (value_diff - median(value_diff)) ./ median(abs(value_diff-median(value_diff))) > opts.ClockResetThresholdOffsetStds & ... 488 | (value_diff - median(value_diff)) > opts.ClockResetThresholdOffsetSeconds; 489 | % points where both a time glitch and a value glitch co-occur are treated as resets 490 | resets_at = time_glitch & value_glitch; 491 | % determine the [begin,end] index ranges between resets 492 | if any(resets_at) 493 | tmp = find(resets_at)'; 494 | tmp = [tmp tmp+1]'; 495 | tmp = [1 tmp(:)' length(resets_at)]; 496 | ranges = num2cell(reshape(tmp,2,[])',2); 497 | if opts.Verbose 498 | disp([' found ' num2str(nnz(resets_at)) ' clock resets in stream ' streams{k}.info.name '.']); end 499 | else 500 | ranges = {[1,length(clock_times)]}; 501 | end 502 | else 503 | % otherwise we just assume that there are no clock resets 504 | ranges = {[1,length(clock_times)]}; 505 | end 506 | 507 | % Calculate clock offset mappings for each data range 508 | mappings = {}; 509 | for r=1:length(ranges) 510 | idx = ranges{r}; 511 | if idx(1) ~= idx(2) 512 | Ax = clock_times(idx(1):idx(2))' / opts.WinsorThreshold; 513 | y = offset_values(idx(1):idx(2))' / opts.WinsorThreshold; 514 | fit_params = robust_fit([ones(size(Ax)) Ax], y); 515 | fit_params(1) = fit_params(1)*opts.WinsorThreshold; 516 | mappings{r} = fit_params; 517 | else 518 | mappings{r} = [offset_values(idx(1)) 0]; % just one measurement 519 | end 520 | end 521 | 522 | if length(ranges) == 1 523 | % apply the correction to all time stamps 524 | temp(k).time_stamps = temp(k).time_stamps + (mappings{1}(1) + mappings{1}(2)*temp(k).time_stamps); 525 | else 526 | % if there are data segments measured with different clocks we need to 527 | % determine, for any time stamp lying between two segments, to which of the segments it belongs 528 | clock_segments = zeros(size(temp(k).time_stamps)); % the segment index to which each stamp belongs 529 | begin_of_segment = 1; % first index into time stamps that belongs to the current segment 530 | end_of_segment = NaN; %#ok % last index into time stamps that belongs to the current segment 531 | for r=1:length(ranges)-1 532 | cur_end_time = clock_times(ranges{r}(2)); % time at which the current segment ends 533 | next_begin_time = clock_times(ranges{r+1}(1)); % time at which the next segment begins 534 | % get the data that is not yet processed 535 | remaining_indices = begin_of_segment:length(temp(k).time_stamps); 536 | if isempty(remaining_indices) 537 | break; end 538 | remaining_data = temp(k).time_stamps(remaining_indices); 539 | if next_begin_time > cur_end_time 540 | % clock jumps forward: the end of the segment is where the data time stamps 541 | % lie closer to the next segment than the current in time 542 | end_of_segment = remaining_indices(min(find([abs(remaining_data-cur_end_time) > abs(remaining_data-next_begin_time),true],1)-1,length(remaining_indices))); 543 | else 544 | % clock jumps backward: the end of the segment is where the data time stamps 545 | % jump back by more than the max conceivable jitter (as any negative delta is jitter) 546 | end_of_segment = remaining_indices(min(find([diff(remaining_data) < -opts.ClockResetMaxJitter,true],1),length(remaining_indices))); 547 | end 548 | % assign the segment of data points to the current range 549 | % go to next segment 550 | clock_segments(begin_of_segment:end_of_segment) = r; 551 | begin_of_segment = end_of_segment+1; 552 | end 553 | % assign all remaining time stamps to the last segment 554 | clock_segments(begin_of_segment:end) = length(ranges); 555 | % apply corrections on a per-segment basis 556 | for r=1:length(ranges) 557 | temp(k).time_stamps(clock_segments==r) = temp(k).time_stamps(clock_segments==r) + (mappings{r}(1) + mappings{r}(2)*temp(k).time_stamps(clock_segments==r)); end 558 | end 559 | end 560 | end 561 | end 562 | 563 | 564 | % =========================================== 565 | % === perform jitter removal if requested === 566 | % =========================================== 567 | if opts.HandleJitterRemoval 568 | % jitter removal is a bonus feature that yields linearly increasing timestamps from data 569 | % where samples had been time stamped with some jitter (e.g., due to operating system 570 | % delays) 571 | if opts.Verbose 572 | disp(' performing jitter removal...'); end 573 | for k=1:length(temp) 574 | if ~isempty(temp(k).time_stamps) && temp(k).srate 575 | 576 | if isfield(streams{k}.info.desc, 'synchronization') && ... 577 | isfield(streams{k}.info.desc.synchronization, 'can_drop_samples') && ... 578 | strcmp(streams{k}.info.desc.synchronization.can_drop_samples, 'true') 579 | temp(k).time_stamps = droppedFramesCorrection(temp(k).time_stamps,temp(k).srate, opts.FrameRateAccuracy); 580 | else 581 | 582 | % identify breaks in the data 583 | diffs = diff(temp(k).time_stamps); 584 | breaks_at = abs(diffs) > max(opts.JitterBreakThresholdSeconds,opts.JitterBreakThresholdSamples*temp(k).sampling_interval); 585 | if any(breaks_at) 586 | % turn the break mask into a cell array of [begin,end] index ranges 587 | tmp = find(breaks_at)'; 588 | tmp = [tmp tmp+1]'; 589 | tmp = [1 tmp(:)' length(breaks_at)]; 590 | ranges = num2cell(reshape(tmp,2,[])',2); 591 | if opts.Verbose 592 | disp([' found ' num2str(nnz(breaks_at)) ' data breaks in stream ' streams{k}.info.name '.']); end 593 | else 594 | ranges = {[1,length(temp(k).time_stamps)]}; 595 | end 596 | 597 | % process each segment separately 598 | segments = repmat(struct(),1,length(ranges)); 599 | for r=1:length(ranges) 600 | range = ranges{r}; 601 | segments(r).num_samples = range(2)-range(1)+1; 602 | segments(r).index_range = range; 603 | if segments(r).num_samples > 0 604 | indices = segments(r).index_range(1):segments(r).index_range(2); 605 | % regress out the jitter 606 | mapping = temp(k).time_stamps(indices) / [ones(1,length(indices)); indices]; 607 | temp(k).time_stamps(indices) = mapping(1) + mapping(2) * indices; 608 | end 609 | % calculate some other meta-data about the segments 610 | segments(r).t_begin = temp(k).time_stamps(range(1)); 611 | segments(r).t_end = temp(k).time_stamps(range(2)); 612 | segments(r).duration = segments(r).t_end - segments(r).t_begin; 613 | segments(r).effective_srate = (segments(r).num_samples - 1) / segments(r).duration; 614 | end 615 | 616 | % calculate the weighted mean sampling rate over all segments 617 | temp(k).effective_rate = sum(bsxfun(@times,[segments.effective_srate],[segments.num_samples]/sum([segments.num_samples]))); 618 | 619 | % transfer the information into the output structs 620 | streams{k}.info.effective_srate = temp(k).effective_rate; 621 | streams{k}.segments = segments; 622 | end 623 | end 624 | end 625 | else 626 | % calculate effective sampling rate 627 | for k=1:length(temp) 628 | if ~isempty(temp(k).time_stamps) 629 | temp(k).effective_srate = (length(temp(k).time_stamps) - 1) / (temp(k).time_stamps(end) - temp(k).time_stamps(1)); 630 | else 631 | temp(k).effective_srate = 0; % BCILAB sets this to NaN 632 | end 633 | % transfer the information into the output structs 634 | streams{k}.info.effective_srate = temp(k).effective_srate; 635 | end 636 | end 637 | 638 | % copy the information into the output 639 | for k=1:length(temp) 640 | offset_mean = 0; 641 | if opts.CorrectStreamLags && ... 642 | isfield(streams{k}.info.desc, 'synchronization') && ... 643 | isfield(streams{k}.info.desc.synchronization, 'offset_mean') 644 | offset_mean = str2num(streams{k}.info.desc.synchronization.offset_mean); 645 | end 646 | streams{k}.time_series = temp(k).time_series; 647 | streams{k}.time_stamps = temp(k).time_stamps - offset_mean; 648 | end 649 | 650 | 651 | % ========================================= 652 | % === peform vendor specific operations === 653 | % ========================================= 654 | 655 | if ~any(strcmp('all',opts.DisableVendorSpecifics)) 656 | 657 | % BrainVision RDA 658 | targetName = 'BrainVision RDA'; 659 | if ~any(strcmp(opts.DisableVendorSpecifics,targetName)) 660 | 661 | % find a target EEG stream... 662 | for k=1:length(streams) 663 | 664 | if strcmp(streams{k}.info.name,targetName) % Is a BrainVision RDA stream? 665 | mkChan = []; 666 | if ~iscell(streams{ k }.info.desc.channels.channel) 667 | warning('Channel structure not a cell array (likely a g.tek writer compatibility issue; Using hack to import channel info)'); 668 | end 669 | for iChan = 1:length( streams{ k }.info.desc.channels.channel ) % Find marker index channel (any channel, not necessary last) 670 | if iscell(streams{ k }.info.desc.channels.channel) 671 | chanStruct = streams{ k }.info.desc.channels.channel{ iChan }; 672 | else 673 | chanStruct = streams{ k }.info.desc.channels.channel( iChan ); 674 | end 675 | if strcmp( chanStruct.label, 'MkIdx' ) && strcmp( chanStruct.type, 'Marker' ) && strcmp( chanStruct.unit, 'Counts (decimal)' ) 676 | mkChan = iChan; 677 | break % Only one marker channel expected 678 | end 679 | end 680 | if ~isempty( mkChan ) % Has a marker index channel? 681 | for m = 1:length( streams ) % find a corresponding indexed marker stream... 682 | if strcmp( streams{ m }.info.name, [ targetName ' Markers' ] ) && strcmp( streams{ m }.info.hostname, streams{ k }.info.hostname ) && strncmp( streams{ m }.info.source_id, streams{ k }.info.source_id, length( streams{ k }.info.source_id ) ) 683 | if opts.Verbose 684 | disp( [ ' performing ', targetName, ' specific tasks for stream ', num2str( k ), '...' ] ); 685 | end 686 | streams = ProcessBVRDAindexedMarkers( streams, k, m, mkChan ); 687 | end 688 | end 689 | 690 | % Remove marker index channel 691 | streams{ k }.time_series( mkChan, : ) = []; 692 | streams{ k }.info.desc.channels.channel( mkChan ) = []; 693 | 694 | % Decrement channel count by 1 695 | streams{ k }.info.channel_count = num2str( str2num( streams{ k }.info.channel_count ) - 1 ); 696 | 697 | end 698 | 699 | end 700 | 701 | end 702 | 703 | end 704 | 705 | end 706 | 707 | 708 | end 709 | 710 | 711 | % ======================== 712 | % === helper functions === 713 | % ======================== 714 | 715 | % scan forward through the file until we find the boundary signature 716 | function scan_forward(f) 717 | block_len = 2^20; 718 | signature = uint8([67 165 70 220 203 245 65 15 179 14 213 70 115 131 203 228]); 719 | while 1 720 | curpos = ftell(f); 721 | block = fread(f,block_len,'*uint8'); 722 | matchpos = strfind(block',signature); 723 | if ~isempty(matchpos) 724 | fseek(f,curpos+matchpos+15,'bof'); 725 | fprintf(' scan forward found a boundary chunk.\n'); 726 | break; 727 | end 728 | if length(block) < block_len 729 | fprintf(' scan forward reached end of file with no match.\n'); 730 | break; 731 | end 732 | end 733 | end 734 | 735 | % read a variable-length integer 736 | function num = read_varlen_int(f) 737 | try 738 | switch fread(f,1,'*uint8') 739 | case 1 740 | num = fread(f,1,'*uint8'); 741 | case 4 742 | num = fread(f,1,'*uint32'); 743 | case 8 744 | num = fread(f,1,'*uint64'); 745 | otherwise 746 | error('Invalid variable-length integer encountered.'); 747 | end 748 | catch %#ok<*CTCH> 749 | num = 0; 750 | end 751 | end 752 | 753 | 754 | % close the file and delete temporary data 755 | function close_file(f,filename) 756 | fclose(f); 757 | if contains(filename, '_temp_uncompressed.xdf') 758 | delete(filename); 759 | end 760 | end 761 | 762 | 763 | % parse a simplified (attribute-free) subset of XML into a MATLAB struct 764 | function result = parse_xml_struct(str) 765 | import org.xml.sax.InputSource 766 | import javax.xml.parsers.* 767 | import java.io.* 768 | tmp = InputSource(); 769 | tmp.setCharacterStream(StringReader(str)); 770 | result = parseChildNodes(xmlread(tmp)); 771 | 772 | % this is part of xml2struct (slightly simplified) 773 | function [children,ptext] = parseChildNodes(theNode) 774 | % Recurse over node children. 775 | children = struct; 776 | ptext = []; 777 | if theNode.hasChildNodes 778 | childNodes = theNode.getChildNodes; 779 | numChildNodes = childNodes.getLength; 780 | for count = 1:numChildNodes 781 | theChild = childNodes.item(count-1); 782 | [text,name,childs] = getNodeData(theChild); 783 | if (~strcmp(name,'#text') && ~strcmp(name,'#comment')) 784 | if (isfield(children,name)) 785 | if (~iscell(children.(name))) 786 | children.(name) = {children.(name)}; end 787 | index = length(children.(name))+1; 788 | children.(name){index} = childs; 789 | if(~isempty(text)) 790 | children.(name){index} = text; end 791 | else 792 | children.(name) = childs; 793 | if(~isempty(text)) 794 | children.(name) = text; end 795 | end 796 | elseif (strcmp(name,'#text')) 797 | if (~isempty(regexprep(text,'[\s]*',''))) 798 | if (isempty(ptext)) 799 | ptext = text; 800 | else 801 | ptext = [ptext text]; 802 | end 803 | end 804 | end 805 | end 806 | end 807 | end 808 | 809 | % this is part of xml2struct (slightly simplified) 810 | function [text,name,childs] = getNodeData(theNode) 811 | % Create structure of node info. 812 | name = char(theNode.getNodeName); 813 | if ~isvarname(name) 814 | name = regexprep(name,'[-]','_dash_'); 815 | name = regexprep(name,'[:]','_colon_'); 816 | name = regexprep(name,'[.]','_dot_'); 817 | end 818 | [childs,text] = parseChildNodes(theNode); 819 | if (isempty(fieldnames(childs))) 820 | try 821 | text = char(theNode.getData); 822 | catch 823 | end 824 | end 825 | end 826 | end 827 | 828 | 829 | function x = robust_fit(A,y,rho,iters) 830 | % Perform a robust linear regression using the Huber loss function. 831 | % x = robust_fit(A,y,rho,iters) 832 | % 833 | % Input: 834 | % A : design matrix 835 | % y : target variable 836 | % rho : augmented Lagrangian variable (default: 1) 837 | % iters : number of iterations to perform (default: 1000) 838 | % 839 | % Output: 840 | % x : solution for x 841 | % 842 | % Notes: 843 | % solves the following problem via ADMM for x: 844 | % minimize 1/2*sum(huber(A*x - y)) 845 | % 846 | % Based on the ADMM Matlab codes also found at: 847 | % https://web.stanford.edu/~boyd/papers/admm_distr_stats.html 848 | % 849 | % Christian Kothe, Swartz Center for Computational Neuroscience, UCSD 850 | % 2013-03-04 851 | 852 | if ~exist('rho','var') 853 | rho = 1; end 854 | if ~exist('iters','var') 855 | iters = 1000; end 856 | 857 | x_offset = min(A(:, 2)); 858 | A(:, 2) = A(:, 2) - x_offset; 859 | Aty = A'*y; 860 | L = chol( A'*A, 'lower' ); 861 | L = sparse(L); 862 | U = sparse(L'); 863 | z = zeros(size(y)); u = z; 864 | for k = 1:iters 865 | x = U \ (L \ (Aty + A'*(z - u))); 866 | d = A*x - y + u; 867 | z = rho/(1+rho)*d + 1/(1+rho)*max(0,(1-(1+1/rho)./abs(d))).*d; 868 | u = d - z; 869 | end 870 | x(1) = x(1) - x(2)*x_offset; 871 | end 872 | 873 | 874 | function res = hlp_superimposedata(varargin) 875 | % Merge multiple partially populated data structures into one fully populated one. 876 | % Result = hlp_superimposedata(Data1, Data2, Data3, ...) 877 | % 878 | % The function is applicable when you have cell arrays or structs/struct arrays with non-overlapping 879 | % patterns of non-empty entries, where all entries should be merged into a single data structure 880 | % which retains their original positions. If entries exist in multiple data structures at the same 881 | % location, entries of later items will be ignored (i.e. earlier data structures take precedence). 882 | % 883 | % In: 884 | % DataK : a data structure that should be super-imposed with the others to form a single data 885 | % structure 886 | % 887 | % Out: 888 | % Result : the resulting data structure 889 | % 890 | % Christian Kothe, Swartz Center for Computational Neuroscience, UCSD 891 | % 2011-08-19 892 | 893 | % first, compactify the data by removing the empty items 894 | compact = varargin(~cellfun('isempty',varargin)); 895 | % start with the last data structure, then merge the remaining data structures into it (in reverse 896 | % order as this avoids having to grow arrays incrementally in typical cases) 897 | res = compact{end}; 898 | for k=length(compact)-1:-1:1 899 | res = merge(res,compact{k}); end 900 | end 901 | 902 | function A = merge(A,B) 903 | % merge data structures A and B 904 | if iscell(A) && iscell(B) 905 | % make sure that both have the same number of dimensions 906 | if ndims(A) > ndims(B) 907 | B = grow_cell(B,size(A)); 908 | elseif ndims(A) < ndims(B) 909 | A = grow_cell(A,size(B)); 910 | end 911 | % make sure that both have the same size 912 | if all(size(B)==size(A)) 913 | % we're fine 914 | elseif all(size(B)>=size(A)) 915 | % A is a minor of B: grow A 916 | A = grow_cell(A,size(B)); 917 | elseif all(size(A)>=size(B)) 918 | % B is a minor of A: grow B 919 | B = grow_cell(B,size(A)); 920 | else 921 | % A and B have mixed sizes... grow both as necessary 922 | M = max(size(A),size(B)); 923 | A = grow_cell(A,M); 924 | B = grow_cell(B,M); 925 | end 926 | % find all non-empty elements in B 927 | idx = find(~cellfun(@(x)isequal(x,[]),B)); 928 | if ~isempty(idx) 929 | % check if any of these is occupied in A 930 | clean = cellfun('isempty',A(idx)); 931 | if ~all(clean) 932 | % merge all conflicting items recursively 933 | conflicts = idx(~clean); 934 | for k=conflicts(:)' 935 | A{k} = merge(A{k},B{k}); end 936 | % and transfer the rest 937 | if any(clean) 938 | A(idx(clean)) = B(idx(clean)); end 939 | else 940 | % transfer all to A 941 | A(idx) = B(idx); 942 | end 943 | end 944 | elseif isstruct(A) && isstruct(B) 945 | % first make sure that both have the same fields 946 | fnA = fieldnames(A); 947 | fnB = fieldnames(B); 948 | if isequal(fnA,fnB) 949 | % we're fine 950 | elseif isequal(sort(fnA),sort(fnB)) 951 | % order doesn't match -- impose A's order on B 952 | B = orderfields(B,fnA); 953 | elseif isempty(setdiff(fnA,fnB)) 954 | % B has a superset of A's fields: add the remaining fields to A, and order them according to B 955 | remaining = setdiff(fnB,fnA); 956 | for fn = remaining' 957 | A(1).(fn{1}) = []; end 958 | A = orderfields(A,fnB); 959 | elseif isempty(setdiff(fnB,fnA)) 960 | % A has a superset of B's fields: add the remaining fields to B, and order them according to A 961 | remaining = setdiff(fnA,fnB); 962 | for fn = remaining' 963 | B(1).(fn{1}) = []; end 964 | B = orderfields(B,fnA); 965 | else 966 | % A and B have incommensurable fields; add B's fields to A's fields, add A's fields to B's 967 | % and order according to A's fields 968 | remainingB = setdiff(fnB,fnA); 969 | for fn = remainingB' 970 | A(1).(fn{1}) = []; end 971 | remainingA = setdiff(fnA,fnB); 972 | for fn = remainingA' 973 | B(1).(fn{1}) = []; end 974 | B = orderfields(B,A); 975 | end 976 | % that being established, convert them to cell arrays, merge their cell arrays, and convert back to structs 977 | merged = merge(struct2cell(A),struct2cell(B)); 978 | A = cell2struct(merged,fieldnames(A),1); 979 | elseif isstruct(A) && ~isstruct(B) 980 | if ~isempty(B) 981 | error('One of the sub-items is a struct, and the other one is of a non-struct type.'); 982 | else 983 | % we retain A 984 | end 985 | elseif isstruct(B) && ~isstruct(A) 986 | if ~isempty(A) 987 | error('One of the sub-items is a struct, and the other one is of a non-struct type.'); 988 | else 989 | % we retain B 990 | A = B; 991 | end 992 | elseif iscell(A) && ~iscell(B) 993 | if ~isempty(B) 994 | error('One of the sub-items is a cell array, and the other one is of a non-cell type.'); 995 | else 996 | % we retain A 997 | end 998 | elseif iscell(B) && ~iscell(A) 999 | if ~isempty(A) 1000 | error('One of the sub-items is a cell array, and the other one is of a non-cell type.'); 1001 | else 1002 | % we retain B 1003 | A = B; 1004 | end 1005 | elseif isempty(A) && ~isempty(B) 1006 | % we retain B 1007 | A = B; 1008 | elseif isempty(B) && ~isempty(A) 1009 | % we retain A 1010 | elseif ~isequal(A,B) 1011 | % we retain A and warn about dropping B 1012 | disp('Two non-empty (and non-identical) sub-elements occupied the same index; one was dropped. This warning will only be displayed once.'); 1013 | end 1014 | end 1015 | 1016 | 1017 | function C = grow_cell(C,idx) 1018 | % grow a cell array to accomodate a particular index 1019 | % (assuming that this index is not contained in the cell array yet) 1020 | tmp = sprintf('%i,',idx); 1021 | eval(['C{' tmp(1:end-1) '} = [];']); 1022 | end 1023 | 1024 | %function written by Matthew Grivich to handle case where frame rate is 1025 | %consistent but with dropped events, like a video camera. 1026 | function frameTimesModeled = droppedFramesCorrection(frameTimes, nominalFrameRate, frameRateAccuracy) 1027 | 1028 | % figure 1029 | % plot(frameTimes(1:end-1), frameTimes(2:end)-frameTimes(1:end-1)); 1030 | % xlabel('Frame Time (s)'); 1031 | % ylabel('Frame Interval (s)'); 1032 | 1033 | 1034 | 1035 | nFrames = length(frameTimes); %nFrames shown, does not included dropped. 1036 | 1037 | %calculates the maximum conceivable number of dropped frames, given the 1038 | %nominal frame rate and the expected frame rate accuracy. 1039 | maxDropped = round((frameTimes(end)-frameTimes(1))*nominalFrameRate*(1+frameRateAccuracy) - nFrames); 1040 | stds = zeros(0, 1); %initialize array of zero length 1041 | for iteration = 1:maxDropped 1042 | interval = (frameTimes(end)-frameTimes(1))/(nFrames-1+iteration-1); 1043 | % fprintf('frequency: %1.20f\n',1/interval); 1044 | frameNumbers = zeros(1,length(frameTimes)); 1045 | 1046 | for i=1:length(frameNumbers) 1047 | 1048 | frameNumbers(i) = round((frameTimes(i)-frameTimes(1))/interval)+1; 1049 | 1050 | end 1051 | %remove zero frame intervals 1052 | for i=length(frameNumbers)-1:-1:1 1053 | if(frameNumbers(i) == frameNumbers(i+1)) 1054 | frameNumbers(i) = frameNumbers(i) -1; 1055 | end 1056 | end 1057 | 1058 | 1059 | pf = polyfit(frameNumbers, frameTimes,1); 1060 | %text(0,.0004, sprintf('interval: %1.20f\n', interval)); 1061 | % fprintf('interval after fit: %1.20f\n',pf(1)); 1062 | 1063 | frameTimesModeled = polyval(pf,frameNumbers); 1064 | % plot(stds) 1065 | % plot(frameTimesModeled, frameTimesModeled - frameTimes); 1066 | % text(0,.02, sprintf('dropped: %d\n', iteration-1)); 1067 | % xlabel('Frame Time (s)') 1068 | % ylabel('Frame Time Modeled - Frame Time (s)'); 1069 | % pause(0.001); 1070 | 1071 | stds(iteration) = std(frameTimesModeled-frameTimes); 1072 | %if (mean(stds) - min(stds))/std(stds) > 10 1073 | % break; 1074 | %end 1075 | 1076 | 1077 | end 1078 | 1079 | % figure 1080 | % plot(stds); 1081 | dropped = find(stds==min(stds)) - 1; 1082 | 1083 | interval = (frameTimes(end)-frameTimes(1))/(nFrames-1+dropped); 1084 | % fprintf('interval: %1.20f\n',interval); 1085 | frameNumbers = zeros(1,length(frameTimes)); 1086 | 1087 | %Find closest frame. 1088 | for i=1:length(frameNumbers) 1089 | frameNumbers(i) = round((frameTimes(i)-frameTimes(1))/interval)+1; 1090 | end 1091 | %remove zero frame intervals 1092 | for i=length(frameNumbers)-1:-1:1 1093 | if(frameNumbers(i) >= frameNumbers(i+1)) 1094 | frameNumbers(i) = frameNumbers(i+1) -1; 1095 | end 1096 | end 1097 | 1098 | 1099 | pf = polyfit(frameNumbers, frameTimes,1); 1100 | %text(0,.0004, sprintf('interval: %1.20f\n', interval)); 1101 | % fprintf('interval after fit: %1.20f\n',pf(1)); 1102 | 1103 | frameTimesModeled = polyval(pf,frameNumbers); 1104 | %figure 1105 | % plot(frameTimesModeled, smooth(frameTimesModeled - frameTimes,21)); 1106 | % xlabel('Frame Time (s)') 1107 | % ylabel('Frame Time Modeled - Frame Time (s)'); 1108 | % pause(0.5); 1109 | 1110 | 1111 | % fprintf('Interval: %1.20f\n',pf(1)); 1112 | % fprintf('Dropped Frames: %d\n', dropped); 1113 | 1114 | 1115 | % figure 1116 | % plot(frameTimes(1:end-1), pf(1)*(frameNumbers(2:end)-frameNumbers(1:end-1))); 1117 | % xlabel('Frame Time (s)'); 1118 | % ylabel('Frame Interval (s)'); 1119 | 1120 | 1121 | end 1122 | 1123 | 1124 | function streams = ProcessBVRDAindexedMarkers( streams, dataStream, mkStream, mkChan ) 1125 | 1126 | clearMarkers = []; 1127 | 1128 | for iMrk = 1:length( streams{ mkStream }.time_series ) 1129 | 1130 | % Decode marker 1131 | MrkInfo = regexp( streams{ mkStream }.time_series{ iMrk }, 'mk(?\d+)=(?.*)', 'names' ); 1132 | 1133 | if ~isempty( MrkInfo.idx ) 1134 | 1135 | % Find corresponding sample in marker index channel 1136 | lat = find( streams{ dataStream }.time_series( mkChan, : ) == str2double( MrkInfo.idx ) ); 1137 | offset = streams{ mkStream }.time_stamps( iMrk ) - streams{ dataStream }.time_stamps( lat ); 1138 | 1139 | % Is the index unique (overflow)? 1140 | if length( lat ) > 1 1141 | [ minOffset, minOffsetIdx ] = min( abs( offset ) ); %#ok 1142 | lat = lat( minOffsetIdx ); 1143 | offset = offset( minOffsetIdx ); 1144 | end 1145 | 1146 | % Sanity check 1147 | if offset > 10 1148 | warning( 'Time stamp difference between indexed marker %s (%.3f s) and corresponding sample in marker channel (%.3f s) exceeding threshold.\n', MrkInfo.idx, streams{ mkStream }.time_stamps( iMrk ), streams{ dataStream }.time_stamps( lat ) ) 1149 | end 1150 | 1151 | % Copy time stamp and rewrite marker 1152 | if ~isempty( lat ) 1153 | streams{ mkStream }.time_stamps( iMrk ) = streams{ dataStream }.time_stamps( lat ); 1154 | streams{ mkStream }.time_series{ iMrk } = MrkInfo.str; 1155 | else 1156 | warning( 'No corresponding sample found in marker channel for indexed marker %s. Removing...', MrkInfo.idx ) 1157 | clearMarkers = [ clearMarkers iMrk ]; 1158 | end 1159 | 1160 | end 1161 | 1162 | end 1163 | 1164 | % Remove markers without corresponding marker channel sample 1165 | streams{ mkStream }.time_stamps( clearMarkers ) = []; 1166 | streams{ mkStream }.time_series( clearMarkers ) = []; 1167 | 1168 | 1169 | end 1170 | --------------------------------------------------------------------------------