├── .github └── workflows │ └── pytest.yml_deactivated_edfapi_missing ├── .gitignore ├── CHANGELOG.md ├── LICENSE ├── Readme.md ├── pyproject.toml ├── setup.cfg ├── setup.py ├── src └── pyedfread │ ├── __init__.py │ ├── data.pxd │ ├── data │ ├── SUB001.EDF │ └── SUB001.asc │ ├── edf_data.pyx │ ├── edf_read.pyx │ └── parse.py ├── tests ├── nppBackup │ ├── test_read_edf.py.2025-04-15_115210.bak │ └── test_read_edf.py.2025-04-15_115222.bak ├── test_read_edf.py └── test_read_sections.py └── tox.ini /.github/workflows/pytest.yml_deactivated_edfapi_missing: -------------------------------------------------------------------------------- 1 | name: Python package 2 | 3 | on: [push] 4 | 5 | jobs: 6 | build: 7 | 8 | runs-on: ubuntu-latest 9 | strategy: 10 | matrix: 11 | python: ["3.8", "3.12"] 12 | 13 | steps: 14 | - uses: actions/checkout@v4 15 | - name: Setup Python 16 | uses: actions/setup-python@v5 17 | with: 18 | python-version: ${{ matrix.python }} 19 | - name: Install tox and any other packages 20 | run: pip install tox 21 | - name: Run tox 22 | # Run tox using the version of Python in `PATH` 23 | run: tox -e py 24 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyc 2 | *.so 3 | *.c 4 | *.o 5 | *.swp 6 | *.pdb 7 | 8 | # Byte-compiled / optimized / DLL files 9 | __pycache__/ 10 | *.py[cod] 11 | 12 | # C extensions 13 | *.so 14 | 15 | # Distribution / packaging 16 | .Python 17 | env/ 18 | build/ 19 | develop-eggs/ 20 | dist/ 21 | downloads/ 22 | eggs/ 23 | .eggs/ 24 | lib/ 25 | lib64/ 26 | parts/ 27 | sdist/ 28 | var/ 29 | *.egg-info/ 30 | .installed.cfg 31 | *.egg 32 | 33 | # PyInstaller 34 | # Usually these files are written by a python script from a template 35 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 36 | *.manifest 37 | *.spec 38 | 39 | # Installer logs 40 | pip-log.txt 41 | pip-delete-this-directory.txt 42 | 43 | # Unit test / coverage reports 44 | htmlcov/ 45 | .tox/ 46 | .coverage 47 | .coverage.* 48 | .cache 49 | nosetests.xml 50 | coverage.xml 51 | *,cover 52 | 53 | # Translations 54 | *.mo 55 | *.pot 56 | 57 | # Django stuff: 58 | *.log 59 | 60 | # Sphinx documentation 61 | docs/_build/ 62 | 63 | # PyBuilder 64 | target/ 65 | 66 | .idea/ 67 | venv/ 68 | -------------------------------------------------------------------------------- /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | # Changelog 2 | 3 | All notable changes to this project will be documented in this file. 4 | 5 | The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), 6 | and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). 7 | ## [0.3.2] - 2025-04-15 8 | ### Major 9 | Richard Vaelle (@flyingfalling) fixed a major, where prior to this release the returned samples dataframe had too many rows (by the number of events), the values in these rows were undefined/unitialized values which, depending on the system, could be 0's, values close to 0, previous memory-usages o, or any other value. This can be very problematic, if by chance the same memory layout is reused, because then the initialized values could be eyetracking data from a previous call to pyedfread. This should be a quite visible error through. 10 | 11 | ### Minor 12 | - Some bugfixes to get tests working on windows 13 | - Some small fixes in the readme 14 | - Bugfix for `parse_msg` where a `startwith` was fixed to `startswith` 15 | 16 | ## [0.3.0] - 2024-07-19 17 | ### Major 18 | So far, blink-events were not explicitly returned. Rather, a `blink` column existed, indicating that within an event, a blink occurred. Two breaking changes: 19 | - `blink` renamed to `contains_blink` 20 | - Blink events are returned explicitly, using `ENDBLINK` event type from Eyelink. I (Benedikt Ehinger) noticed that the ENDBLINK and STARTBLINK start-times differed by 1 sample, but I have no explanation for this and just use the `ENDBLINK` data. 21 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2024, Niklas Wilming, Neal Morton, Benedikt Ehinger 2 | All rights reserved. 3 | 4 | Redistribution and use in source and binary forms, with or without 5 | modification, are permitted provided that the following conditions are met: 6 | 7 | 1. Redistributions of source code must retain the above copyright notice, this 8 | list of conditions and the following disclaimer. 9 | 2. Redistributions in binary form must reproduce the above copyright notice, 10 | this list of conditions and the following disclaimer in the documentation 11 | and/or other materials provided with the distribution. 12 | 13 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND 14 | ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 15 | WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 16 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR 17 | ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES 18 | (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 19 | LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND 20 | ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 21 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 22 | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 23 | -------------------------------------------------------------------------------- /Readme.md: -------------------------------------------------------------------------------- 1 | # pyedfread 2 | 3 | 4 | A utility that parses SR research EDF data files into pandas DataFrames. 5 | This package was initially developed by [Niklas Wilming](https://github.com/nwilming/) and updated by [Neal Morton](https://github.com/mortonne/) under the name of `edfread`. Now it is maintained by the [Computational Cognitive Science Stuttgart](www.s-ccs.de) group with official permission from Wilming & Morton. 6 | 7 | ## Requirements 8 | 9 | 10 | EyeLink Developers Kit. Download from [SR-Research support forum](https://www.sr-research.com/support/thread-13.html) 11 | (forum registration required). 12 | 13 | > I do not include the SR Research EDF Access API header files and libraries. 14 | > These are needed to compile pyedfread with cython. If you use a mac you can 15 | > download a package from the SR-Research support forum. If you use Ubuntu you 16 | > can install them via apt-get. 17 | 18 | 19 | ## Alternatives 20 | Due to the "unmaintained" status of this repo ca. 2019-2024 some alternatives came up: 21 | 22 | [eyelinkio](https://github.com/scott-huberty/eyelinkio) 23 | 24 | If you are willing to run `edf2asc` on all your files, you can use: 25 | 26 | [mne-python](https://mne.tools/stable/auto_tutorials/io/70_reading_eyetracking_data.html) 27 | 28 | [ParseEyeLinkAscFiles](https://github.com/djangraw/ParseEyeLinkAscFiles) 29 | 30 | [eyelinkparser](https://github.com/open-cogsci/eyelinkparser) 31 | 32 | ## Contributors 33 | - Niklas Wilming 34 | - Selim Onat 35 | - Chadwick Boulay 36 | - Niel Morton 37 | - Andrea Constantino 38 | - Benedikt Ehinger (maintainer) 39 | - Alex Kim 40 | - Greg Schwimer 41 | - Oscar Esteban 42 | - Richard Veale 43 | 44 | ## Setup 45 | 46 | Run `pip install git+https://github.com/s-ccs/pyedfread` to compile and install. This will install the 47 | python library and a command line script to parse edfs. 48 | 49 | 50 | ### Windows Support 51 | As of this writing (July 2019), the EyeLink Developers Kit for Windows requires small modifications 52 | before it will work with this project. Administrator access is required to edit the files 53 | in their default directory: C:\Program Files (x86)\SR Research\EyeLink\Includes\eyelink/ 54 | 55 | Edit edftypes.h. Replace the chunk of typedefs (lines 40-49) with the following: 56 | ```C 57 | #ifndef BYTEDEF 58 | #define BYTEDEF 1 59 | typedef unsigned char byte; 60 | #ifndef _BASETSD_H_ /* windows header */ 61 | typedef short INT16; 62 | typedef int INT32; 63 | typedef unsigned short UINT16; 64 | typedef unsigned int UINT32; 65 | #endif 66 | #endif 67 | ``` 68 | 69 | Some user reported the following, but in 2024 BenediktEhinger didnt need this: 70 | 71 | One of the DLLs from SR Research (zlibwapi.dll) depends on MSVCP90.dll. 72 | This _should_ come with MS Visual Studio C++ 2008 SP1 redistributable, but its installer didn't seem to put the file on the PATH. 73 | Instead, I had already installed [Mercurial for Windows](https://www.mercurial-scm.org/release/windows/mercurial-4.9.1-x64.msi) 74 | which comes with the correct version of that file and puts it on the PATH by default. 75 | 76 | 77 | ## Usage 78 | 79 | pyedfread can be used on the command line (convert_edf) or called from 80 | within python. 81 | 82 | ## From python 83 | 84 | After compilation run the following lines for a quick test. 85 | 86 | >>> import pyedfread 87 | >>> samples, events, messages = pyedfread.read_edf('SUB001.EDF') 88 | 89 | This opens SUB001.EDF and parses it three DataFrames: 90 | 91 | - samples contain individual samples. 92 | - events contains fixation and saccade definitions 93 | - messages contains meta data associated with each trial. 94 | 95 | pyedfread allows to select which meta data you want to read from your edf file. 96 | This happens through the 'filter' argument of edf.pread / pyedfread.fread. It can 97 | contain a list of 'message' identifiers. A message identifier in the EDF is 98 | trial metadata injected into the data stream during eye tracking. If 99 | for example after the start of a trial you send the message "condition 1", you 100 | can add 'condition' to the filter list and pyedfread will automagically add a 101 | field condition with value 1 for each trial to the messages structure. Of course, 102 | if the value varies across trials this will be reflected in the messages 103 | structure. This is what it looks like in python code: 104 | 105 | >>> samples, events, messages = edf.read_edf('SUB001.EDF', ignore_samples=True, message_filter=['condition']) 106 | 107 | If the filter is not specified, pyedfread saves all messages it can parse. 108 | 109 | The column names map almost directly to C structure names in the EDF C API. To 110 | understand column content check the edf acces api documentation (2.1.1 FSAMPLE 111 | Structure + 2.1.2 FEVENT Struct.) and see the examples listed below. Column names 112 | for each eye are suffixed by 'left' or 'right' respectively. 113 | 114 | Some examples for samples: 115 | 116 | - time: time stamp of sample 117 | - flags: flags to indicate contents 118 | - px: pupil x, y 119 | - py 120 | - hx: headref x, y 121 | - hy 122 | - pa: pupil size / area 123 | - gx: screen gaze x, y 124 | - gy 125 | - rx: screen pixels per degree 126 | - ry 127 | - status: status flags 128 | - input: extra input word 129 | - buttons: button state & change 130 | - htype: head-tracker data 131 | - hdata: Not included 132 | - errors: process error flags 133 | - gxvel: gaze x,y velocity 134 | - gyvel 135 | - hxvel: headref x,y velocity 136 | - hyvel 137 | - rxvel: raw x, y velocity 138 | - ryvel 139 | - fgxvel: fast gaze x, y velocity 140 | - fgyvel 141 | - fhxvel: fast headref x, y velocity 142 | - fhyvel 143 | - frxvel: fast raw x, y velocity 144 | - fryvel 145 | 146 | Some examples for events: 147 | 148 | - time - time point where event occured 149 | - type - event type 150 | - eye - eye: 0=left, 1=right 151 | - start - start time 152 | - end - end time 153 | - gavx, gavy - average gaze location 154 | 155 | 156 | ## Command line 157 | 158 | 159 | If you have an EDF file with "standard" meta data (e.g. key and value are seperated by a 160 | blank) you can call 161 | 162 | $> convert_edf SUB001.EDF sub001.hdf 163 | 164 | The .hdf file is a valid hdf5 file that can be read into matlab. The default is 165 | to simplify the HDF file by replacing strings with numbers. The original strings 166 | are saved as attributes in the HDF file. Fields that contain python objects (e.g. 167 | arrays, lists etc.) are skipped. 168 | 169 | Run 'read_edf -h' for some help. 170 | 171 | 172 | ## Matlab 173 | 174 | 175 | To read a datamat into matlab do this: 176 | 177 | >> info = h5info('SUB001.hdf'); 178 | >> info.Groups.Datasets 179 | >> field = h5read('SUB001.hdf', '/events/gavx'); 180 | 181 | 182 | ## Testing 183 | While there are some basic unit-testing setup, it is currently not possible to use continuous integration, as we cannot provide the required SR-Research libraries. 184 | 185 | if you want to run the test use: 186 | 187 | `tox -e py` 188 | 189 | after cloning 190 | 191 | ## License 192 | 193 | 194 | BSD License, see LICENSE file 195 | -------------------------------------------------------------------------------- /pyproject.toml: -------------------------------------------------------------------------------- 1 | [build-system] 2 | requires = ["setuptools", "wheel", "numpy", "cython"] 3 | build-backend = "setuptools.build_meta" 4 | 5 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [metadata] 2 | name = pyedfread 3 | version = 0.3.2 4 | description = Read SR-Research EDF files. 5 | keywords = psychology, neuroscience 6 | license = BSD-2-Clause 7 | classifiers = 8 | Programming Language :: Python :: 3 9 | Operating System :: OS Independent 10 | 11 | [options] 12 | install_requires = 13 | numpy 14 | pandas 15 | h5py 16 | package_dir = src 17 | packages = find: 18 | 19 | [options.package_data] 20 | pyedfread = data/*.asc, data/*.EDF 21 | 22 | [options.packages.find] 23 | where = src 24 | 25 | [options.extras_require] 26 | test = pytest 27 | 28 | [options.entry_points] 29 | console_scripts = 30 | convert_edf = pyedfread:parse.convert_edf 31 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """ 3 | Setup file for pyedf. Builds cython extension needed for using 4 | the C EDF access API. 5 | 6 | To build locally call 7 | python setup.py build_ext --inplace 8 | 9 | """ 10 | 11 | import sys 12 | #from distutils.core import setup 13 | #from distutils.extension import Extension 14 | from Cython.Distutils import build_ext 15 | from setuptools import setup,find_packages, Extension 16 | import numpy 17 | 18 | 19 | if sys.platform.startswith("darwin"): 20 | args = { 21 | "include_dirs": [ 22 | numpy.get_include(), 23 | "/Library/Frameworks/edfapi.framework/Headers", 24 | ], 25 | "extra_link_args": ["-F/Library/Frameworks/", "-framework", "edfapi"], 26 | "extra_compile_args": ["-w"], 27 | } 28 | elif sys.platform.startswith("win32"): 29 | import os 30 | import platform 31 | 32 | srr_basedir = r"C:\Program Files (x86)\SR Research\EyeLink" 33 | if platform.architecture()[0] == "64bit": 34 | arch_dir = 'x64' 35 | lib_names = ['edfapi64'] 36 | else: 37 | arch_dir = '' 38 | lib_names = ['edfapi'] 39 | args = {'include_dirs': [numpy.get_include(), os.path.join(srr_basedir, 'Includes'),os.path.join(srr_basedir, 'Includes',"eyelink")], 40 | 'library_dirs': [os.path.join(srr_basedir, 'libs', arch_dir)], 41 | 'libraries': lib_names 42 | } 43 | 44 | else: # linux, unix, cygwin 45 | args = { 46 | "include_dirs": [numpy.get_include(), "include/", '/usr/include/EyeLink/'], 47 | "library_dirs": ["lib/"], 48 | "libraries": ["edfapi"], 49 | "extra_compile_args": ["-fopenmp"], 50 | "extra_link_args": ["-fopenmp"], 51 | } 52 | 53 | print(args) 54 | ext_module = Extension("pyedfread.edf_read", ["src/pyedfread/edf_read.pyx"], **args) 55 | 56 | ext_data = Extension("pyedfread.edf_data", ["src/pyedfread/edf_data.pyx"], **args) 57 | 58 | setup( 59 | cmdclass={'build_ext': build_ext}, 60 | ext_modules=[ext_data, ext_module], 61 | packages = find_packages(where='src'), 62 | package_dir = {"": "src"}, 63 | ) 64 | -------------------------------------------------------------------------------- /src/pyedfread/__init__.py: -------------------------------------------------------------------------------- 1 | """Read eye-tracking information from EyeLink EDF files.""" 2 | 3 | 4 | import sys 5 | if sys.platform.startswith("win32"): 6 | import os 7 | import platform 8 | srr_basedir = r"C:\Program Files (x86)\SR Research\EyeLink" 9 | if platform.architecture()[0] == "64bit": 10 | arch_dir = 'x64' 11 | else: 12 | arch_dir = '' 13 | # due to a change in python 3.8, the searchpath is not automatically added anymore 14 | os.add_dll_directory(os.path.join(srr_basedir, 'libs', arch_dir)) 15 | 16 | from pyedfread.parse import read_edf 17 | from pyedfread.edf_read import read_preamble, read_messages, read_calibration 18 | -------------------------------------------------------------------------------- /src/pyedfread/data.pxd: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """Define SR Research structures used in the EDF.""" 3 | 4 | from libc.stdint cimport int16_t, uint16_t, uint32_t, int64_t 5 | 6 | cdef extern from 'edf_data.h': 7 | ctypedef struct FSAMPLE: 8 | uint32_t time # time of sample 9 | float px[2] 10 | float py[2] # pupil xy 11 | float hx[2] 12 | float hy[2] # headref xy 13 | float pa[2] # pupil size or area 14 | float gx[2] 15 | float gy[2] # screen gaze xy 16 | float rx 17 | float ry # screen pixels per degree 18 | 19 | float gxvel[2] 20 | float gyvel[2] 21 | float hxvel[2] 22 | float hyvel[2] 23 | float rxvel[2] 24 | float ryvel[2] 25 | 26 | float fgxvel[2] 27 | float fgyvel[2] 28 | float fhxvel[2] 29 | float fhyvel[2] 30 | float frxvel[2] 31 | float fryvel[2] 32 | 33 | int16_t hdata[8] # head-tracker data (not prescaled) 34 | uint16_t flags # flags to indicate contents 35 | uint16_t input # extra (input word) 36 | uint16_t buttons # button state & changes 37 | int16_t htype # head-tracker data type 38 | int16_t errors # process error flags 39 | 40 | ctypedef struct LSTRING: 41 | int16_t len 42 | char c 43 | 44 | ctypedef struct FEVENT: 45 | uint32_t time # effective time of event 46 | int16_t type # event type 47 | int16_t read # flags which items were included 48 | int16_t eye # eye: 0=left,1=right 49 | 50 | uint32_t sttime, entime # start, end times 51 | float hstx, hsty # starting points 52 | float gstx, gsty # starting points 53 | float sta 54 | float henx, heny # ending points 55 | float genx, geny # ending points 56 | float ena 57 | float havx, havy # averages 58 | float gavx, gavy # averages 59 | float ava 60 | float avel # avg velocity accum 61 | float pvel # peak velocity accum 62 | float svel, evel # start, end velocity 63 | float supd_x, eupd_x # start, end units-per-degree 64 | float supd_y, eupd_y # start, end units-per-degree 65 | 66 | uint16_t status # error, warning flags 67 | uint16_t flags # error, warning flag 68 | uint16_t input 69 | uint16_t buttons 70 | uint16_t parsedby # 7 bits of flags: PARSEDBY code 71 | LSTRING *message # any message strin 72 | 73 | ctypedef struct IMESSAGE: 74 | uint32_t time # time message logged 75 | int16_t type # event type: usually MESSAGEEVENT 76 | uint16_t length # length of message 77 | char text[260] # message contents (max length 255) 78 | 79 | ctypedef struct IOEVENT: 80 | uint32_t time # time logged 81 | int16_t type # event type 82 | uint16_t data # coded event data 83 | 84 | ctypedef struct RECORDINGS: 85 | uint32_t time # start time or end tim 86 | float sample_rate # 250 or 500 or 100 87 | uint16_t eflags # to hold extra information about events 88 | uint16_t sflags # to hold extra information about samples 89 | char state # 0 = END, 1 = START 90 | char record_type # 1 = SAMPLES, 2 = EVENTS, 3 = SAMPLES and EVENT 91 | char pupil_type # 0 = AREA, 1 = DIAMETE 92 | char recording_mode # 0 = PUPIL, 1 = CR 93 | char filter_type # 1, 2, 3 94 | char pos_type # PARSEDBY_GAZE PARSEDBY_HREF PARSEDBY_PUPI 95 | char eye # 1 = LEFT, 2 = RIGHT, 3 = LEFT and RIGHT 96 | 97 | ctypedef union ALLF_DATA: 98 | FEVENT fe 99 | IMESSAGE im 100 | IOEVENT io 101 | FSAMPLE fs 102 | RECORDINGS rec 103 | -------------------------------------------------------------------------------- /src/pyedfread/data/SUB001.EDF: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/s-ccs/pyedfread/4a8800cc003b13075d032cbaf33f076b7ee39aaf/src/pyedfread/data/SUB001.EDF -------------------------------------------------------------------------------- /src/pyedfread/edf_data.pyx: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """Read SR Research EDF files and parse them into ocupy datamats.""" 3 | 4 | 5 | MISSING_DATA = -32768 # data is missing (integer) 6 | MISSING = -32768 7 | INaN = -32768 8 | 9 | LEFT_EYE = 0 # index and ID of eyes 10 | RIGHT_EYE = 1 11 | LEFTEYEI = 0 12 | RIGHTEYEI = 1 13 | LEFT = 0 14 | RIGHT = 1 15 | 16 | BINOCULAR = 2 # data for both eyes available 17 | 18 | # The SAMPLE struct contains data from one 4-msec eye-tracker sample. 19 | # The field has a bit for each type of data in the sample. 20 | # Fields not read have 0 flag bits, and are set to MISSING_DATA. 21 | 22 | SAMPLE_LEFT = 0x8000 # data for these eye(s) 23 | SAMPLE_RIGHT = 0x4000 24 | 25 | SAMPLE_TIMESTAMP = 0x2000 # always for link, used to compress files 26 | 27 | SAMPLE_PUPILXY = 0x1000 # pupil x, y pair 28 | SAMPLE_HREFXY = 0x0800 # head-referenced x, y pair 29 | SAMPLE_GAZEXY = 0x0400 # gaze x, y pair 30 | SAMPLE_GAZERES = 0x0200 # gaze res (x, y pixels per degree) pair 31 | SAMPLE_PUPILSIZE = 0x0100 # pupil size 32 | SAMPLE_STATUS = 0x0080 # error flags 33 | SAMPLE_INPUTS = 0x0040 # input data port 34 | SAMPLE_BUTTONS = 0x0020 # button state: LSBy state, MSBy changes 35 | 36 | SAMPLE_HEADPOS = 0x0010 # head-position: byte tells # words 37 | SAMPLE_TAGGED = 0x0008 # reserved variable-length tagged 38 | SAMPLE_UTAGGED = 0x0004 # user-defineabe variable-length tagged 39 | SAMPLE_ADD_OFFSET = 0x0002 # if this flag is set for the sample add .5ms to the sample time 40 | 41 | FSAMPLEDEF = 1 # gaze, resolution prescaling removed 42 | FEVENTDEF = 1 43 | 44 | SAMPLE_TYPE = 200 45 | 46 | STARTPARSE = 1 # these only have time and eye data 47 | ENDPARSE = 2 48 | BREAKPARSE = 10 49 | 50 | # EYE DATA: contents determined by evt_data 51 | STARTBLINK = 3 # and by "read" data item 52 | ENDBLINK = 4 # all use IEVENT format 53 | STARTSACC = 5 54 | ENDSACC = 6 55 | STARTFIX = 7 56 | ENDFIX = 8 57 | FIXUPDATE = 9 58 | 59 | STARTSAMPLES = 15 # start of events in block 60 | ENDSAMPLES = 16 # end of samples in block 61 | STARTEVENTS = 17 # start of events in block 62 | ENDEVENTS = 18 # end of events in block 63 | 64 | MESSAGEEVENT = 24 # user-definable text or data 65 | BUTTONEVENT = 25 # button state change 66 | INPUTEVENT = 28 # change of input port 67 | 68 | LOST_DATA_EVENT = 0x3F # NEW: Event flags gap in data stream 69 | 70 | READ_ENDTIME = 0x0040 # end time (start time always read) 71 | READ_GRES = 0x0200 # gaze resolution xy 72 | READ_SIZE = 0x0080 # pupil size 73 | READ_VEL = 0x0100 # velocity (avg, peak) 74 | READ_STATUS = 0x2000 # status (error word) 75 | 76 | READ_BEG = 0x0001 # event has start data for vel, size, gres 77 | READ_END = 0x0002 # event has end data for vel, size, gres 78 | READ_AVG = 0x0004 # event has avg pupil size, velocity 79 | 80 | # position eye data 81 | READ_PUPILXY = 0x0400 # pupilxy REPLACES gaze, href data if read 82 | READ_HREFXY = 0x0800 83 | READ_GAZEXY = 0x1000 84 | 85 | READ_BEGPOS = 0x0008 # position data for these parts of event 86 | READ_ENDPOS = 0x0010 87 | READ_AVGPOS = 0x0020 88 | 89 | # RAW FILE/LINK CODES: REVERSE IN R/W 90 | FRIGHTEYE_EVENTS = 0x8000 # has right eye events 91 | FLEFTEYE_EVENTS = 0x4000 # has left eye events 92 | 93 | # "event_types" flag in ILINKDATA or EDF_FILE 94 | # tells what types of events were written by tracker 95 | 96 | LEFTEYE_EVENTS = 0x8000 # has left eye events 97 | RIGHTEYE_EVENTS = 0x4000 # has right eye events 98 | BLINK_EVENTS = 0x2000 # has blink events 99 | FIXATION_EVENTS = 0x1000 # has fixation events 100 | FIXUPDATE_EVENTS = 0x0800 # has fixation updates 101 | SACCADE_EVENTS = 0x0400 # has saccade events 102 | MESSAGE_EVENTS = 0x0200 # has message events 103 | BUTTON_EVENTS = 0x0040 # has button events 104 | INPUT_EVENTS = 0x0020 # has input port events 105 | 106 | # "event_data" flags in ILINKDATA or EDF_FILE 107 | # tells what types of data were included in events by tracker 108 | 109 | EVENT_VELOCITY = 0x8000 # has velocity data 110 | EVENT_PUPILSIZE = 0x4000 # has pupil size data 111 | EVENT_GAZERES = 0x2000 # has gaze resolution 112 | EVENT_STATUS = 0x1000 # has status flags 113 | 114 | EVENT_GAZEXY = 0x0400 # has gaze xy position 115 | EVENT_HREFXY = 0x0200 # has head-ref xy position 116 | EVENT_PUPILXY = 0x0100 # has pupil xy position 117 | 118 | FIX_AVG_ONLY = 0x0008 # only avg. data to fixation evts 119 | START_TIME_ONLY = 0x0004 # only start-time in start events 120 | 121 | PARSEDBY_GAZE = 0x00C0 # how events were generated 122 | PARSEDBY_HREF = 0x0080 123 | PARSEDBY_PUPIL = 0x0040 124 | 125 | LED_TOP_WARNING = 0x0080 # marker is in border of image 126 | LED_BOT_WARNING = 0x0040 127 | LED_LEFT_WARNING = 0x0020 128 | LED_RIGHT_WARNING = 0x0010 129 | HEAD_POSITION_WARNING = 0x00F0 # head too far from calibr??? 130 | 131 | LED_EXTRA_WARNING = 0x0008 # glitch or extra markers 132 | LED_MISSING_WARNING = 0x0004 # <2 good data points in last 100 msec) 133 | HEAD_VELOCITY_WARNING = 0x0001 # head moving too fast 134 | 135 | CALIBRATION_AREA_WARNING = 0x0002 # pupil out of good mapping area 136 | 137 | MATH_ERROR_WARNING = 0x2000 # math error in proc. sample 138 | 139 | INTERP_SAMPLE_WARNING = 0x1000 140 | INTERP_PUPIL_WARNING = 0x8000 141 | 142 | CR_WARNING = 0x0F00 143 | CR_LEFT_WARNING = 0x0500 144 | CR_RIGHT_WARNING = 0x0A00 145 | 146 | CR_LOST_WARNING = 0x0300 147 | CR_LOST_LEFT_WARNING = 0x0100 148 | CR_LOST_RIGHT_WARNING = 0x0200 149 | 150 | CR_RECOV_WARNING = 0x0C00 151 | CR_RECOV_LEFT_WARNING = 0x0400 152 | CR_RECOV_RIGHT_WARNING = 0x0800 153 | 154 | TFLAG_MISSING = 0x4000 # missing 155 | TFLAG_ANGLE = 0x2000 # extreme target angle 156 | TFLAG_NEAREYE = 0x1000 # target near eye so windows overlapping 157 | # DISTANCE WARNINGS (limits set by remote_distance_warn_range command) 158 | TFLAG_CLOSE = 0x0800 # distance vs. limits 159 | TFLAG_FAR = 0x0400 160 | # TARGET TO CAMERA EDGE (margin set by remote_edge_warn_pixels command) 161 | TFLAG_T_TSIDE = 0x0080 # target near edge of image (left, right, top, bottom) 162 | TFLAG_T_BSIDE = 0x0040 163 | TFLAG_T_LSIDE = 0x0020 164 | TFLAG_T_RSIDE = 0x0010 165 | # EYE TO CAMERA EDGE (margin set by remote_edge_warn_pixels command) 166 | TFLAG_E_TSIDE = 0x0008 # eye near edge of image (left, right, top, bottom) 167 | TFLAG_E_BSIDE = 0x0004 168 | TFLAG_E_LSIDE = 0x0002 169 | TFLAG_E_RSIDE = 0x0001 170 | 171 | NO_PENDING_ITEMS = 0 172 | RECORDING_INFO = 30 173 | 174 | PUPIL_CR = 2 175 | PUPIL_ONLY_500 = 1 176 | PUPIL_ONLY_250 = 0 177 | -------------------------------------------------------------------------------- /src/pyedfread/edf_read.pyx: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # cython: profile=True 3 | """Read SR Research EDF files and parse them into a list of dicts.""" 4 | 5 | cimport numpy as np 6 | import numpy as np 7 | import string 8 | 9 | from libc.stdint cimport int16_t, uint16_t, uint32_t, int64_t 10 | from libc.stdlib cimport malloc, free 11 | 12 | from libc.stdio cimport printf 13 | 14 | from pyedfread.edf_data import * 15 | from pyedfread.data cimport ALLF_DATA 16 | 17 | import struct 18 | 19 | 20 | type2label = { 21 | STARTFIX: 'fixation', 22 | STARTSACC: 'saccade', 23 | STARTBLINK: 'blink', 24 | ENDFIX: 'fixation', 25 | ENDSACC: 'saccade', 26 | ENDBLINK: 'blink', 27 | MESSAGEEVENT: 'message', 28 | } 29 | 30 | sample_columns = [ 31 | 'time', 32 | 'px_left', 33 | 'px_right', 34 | 'py_left', 35 | 'py_right', 36 | 'hx_left', 37 | 'hx_right', 38 | 'hy_left', 39 | 'hy_right', 40 | 'pa_left', 41 | 'pa_right', 42 | 'gx_left', 43 | 'gx_right', 44 | 'gy_left', 45 | 'gy_right', 46 | 'rx', 47 | 'ry', 48 | 'gxvel_left', 49 | 'gxvel_right', 50 | 'gyvel_left', 51 | 'gyvel_right', 52 | 'hxvel_left', 53 | 'hxvel_right', 54 | 'hyvel_left', 55 | 'hyvel_right', 56 | 'rxvel_left', 57 | 'rxvel_right', 58 | 'ryvel_left', 59 | 'ryvel_right', 60 | 'fgxvel', 61 | 'fgyvel', 62 | 'fhxvel', 63 | 'fhyvel', 64 | 'frxvel', 65 | 'fryvel', 66 | 'flags', 67 | 'input', 68 | 'buttons', 69 | 'htype', 70 | 'errors', 71 | ] 72 | 73 | 74 | cdef extern from "edf.h": 75 | ctypedef int EDFFILE 76 | EDFFILE * edf_open_file( 77 | const char * fname, 78 | int consistency, 79 | int load_events, 80 | int load_samples, 81 | int * errval, 82 | ) 83 | int edf_get_preamble_text_length(EDFFILE * edf) 84 | int edf_get_preamble_text(EDFFILE * ef, char * buffer, int length) 85 | int edf_get_next_data(EDFFILE * ef) 86 | ALLF_DATA * edf_get_float_data(EDFFILE * ef) 87 | int edf_get_element_count(EDFFILE * ef) 88 | int edf_close_file(EDFFILE * ef) 89 | 90 | 91 | def read_preamble(filename, consistency=0): 92 | """Read preamble of EDF file.""" 93 | cdef int errval = 1 94 | cdef EDFFILE * ef 95 | ef = edf_open_file(filename.encode('utf-8'), consistency, 1, 1, & errval) 96 | if errval < 0: 97 | raise IOError(f'Could not open file: {filename}') 98 | cdef int psize = edf_get_preamble_text_length(ef) 99 | cdef char * buf = malloc(psize * sizeof(char)) 100 | e = edf_get_preamble_text(ef, buf, psize) 101 | edf_close_file(ef) 102 | return buf.decode('utf-8') 103 | 104 | 105 | def read_messages(filename, startswith=None, consistency=0): 106 | """Read messages from an edf file.""" 107 | cdef int errval = 1 108 | cdef EDFFILE * ef 109 | cdef char * msg 110 | ef = edf_open_file(filename.encode('utf-8'), consistency, 1, 1, & errval) 111 | if errval < 0: 112 | raise IOError(f'Could not open file: {filename}') 113 | 114 | messages = [] 115 | samples = [] 116 | while True: 117 | sample_type = edf_get_next_data(ef) 118 | if sample_type == NO_PENDING_ITEMS: 119 | edf_close_file(ef) 120 | break 121 | samples.append(sample_type) 122 | if sample_type == MESSAGEEVENT or sample_type == RECORDING_INFO: 123 | fd = edf_get_float_data(ef) 124 | message = '' 125 | if < int > fd.fe.message != 0: 126 | msg = &fd.fe.message.c 127 | message = msg[:fd.fe.message.len] 128 | message = message.decode('utf-8').replace('\x00', '').strip() 129 | if ( 130 | startswith is None 131 | or any([message.startswith(s) for s in startswith]) 132 | ): 133 | messages.append(message) 134 | return messages 135 | 136 | 137 | def read_calibration(filename, consistency=0): 138 | """Read calibration/validation messages from an EDF file.""" 139 | start = ['!VAL', '!CAL'] 140 | messages = read_messages(filename, startswith=start, consistency=consistency) 141 | return messages 142 | 143 | 144 | cdef data2dict(sample_type, EDFFILE * ef): 145 | """Convert EDF event to a dictionary.""" 146 | fd = edf_get_float_data(ef) 147 | cdef char * msg 148 | d = None 149 | if ( 150 | (sample_type == STARTFIX) 151 | or (sample_type == STARTSACC) 152 | or (sample_type == STARTBLINK) 153 | or (sample_type == ENDFIX) 154 | or (sample_type == ENDSACC) 155 | or (sample_type == ENDBLINK) 156 | or (sample_type == MESSAGEEVENT) 157 | ): 158 | message = '' 159 | if < int > fd.fe.message != 0: 160 | msg = & fd.fe.message.c 161 | message = msg[:fd.fe.message.len] 162 | d = { 163 | 'time': fd.fe.time, 164 | 'type': type2label[sample_type], 165 | 'start': fd.fe.sttime, 166 | 'end': fd.fe.entime, 167 | 'hstx': fd.fe.hstx, 168 | 'hsty': fd.fe.hsty, 169 | 'gstx': fd.fe.gstx, 170 | 'gsty': fd.fe.gsty, 171 | 'sta': fd.fe.sta, 172 | 'henx': fd.fe.henx, 173 | 'heny': fd.fe.heny, 174 | 'genx': fd.fe.genx, 175 | 'geny': fd.fe.geny, 176 | 'ena': fd.fe.ena, 177 | 'havx': fd.fe.havx, 178 | 'havy': fd.fe.havy, 179 | 'gavx': fd.fe.gavx, 180 | 'gavy': fd.fe.gavy, 181 | 'ava': fd.fe.ava, 182 | 'avel': fd.fe.avel, 183 | 'pvel': fd.fe.pvel, 184 | 'svel': fd.fe.svel, 185 | 'evel': fd.fe.evel, 186 | 'supd_x': fd.fe.supd_x, 187 | 'eupd_x': fd.fe.eupd_x, 188 | 'eye': fd.fe.eye, 189 | 'buttons': fd.fe.buttons, 190 | 'message': message, 191 | } 192 | return d 193 | 194 | 195 | def parse_datum( 196 | data, 197 | sample_type, 198 | trial, 199 | current_event, 200 | event_accumulator, 201 | ): 202 | """Parse a datum into data structures.""" 203 | if data is None: 204 | return current_event 205 | 206 | if (sample_type == STARTFIX) or (sample_type == STARTSACC): 207 | current_event = data 208 | current_event['contains_blink'] = False 209 | current_event['trial'] = trial 210 | 211 | 212 | if (sample_type == ENDFIX) or (sample_type == ENDSACC): 213 | current_event.update(data) 214 | event_accumulator.append(current_event) 215 | 216 | if (sample_type == STARTBLINK) or (sample_type == ENDBLINK): 217 | current_event['contains_blink'] = True 218 | if (sample_type == ENDBLINK): 219 | event_accumulator.append(data) 220 | 221 | return current_event 222 | 223 | 224 | def parse_message(data, trial, message_accumulator, message_filter, trial_marker): 225 | """Parse message information based on message type.""" 226 | message = data['message'].decode('utf-8').replace('\x00', '').strip() 227 | if message.startswith(trial_marker): 228 | if trial <= 0: 229 | trial = 1 230 | else: 231 | trial += 1 232 | 233 | if message_filter is None or any([message.startswith(s) for s in message_filter]): 234 | info = {'time': data['start'], 'trial': trial, 'message': message} 235 | message_accumulator.append(info) 236 | return trial 237 | 238 | 239 | def parse_edf( 240 | filename, ignore_samples=False, message_filter=None, trial_marker='TRIALID' 241 | ): 242 | """Read samples, events, and messages from an EDF file.""" 243 | cdef int errval = 1 244 | cdef char * buf = < char * > malloc(1024 * sizeof(char)) 245 | cdef EDFFILE * ef 246 | cdef int sample_type, cnt, trial 247 | 248 | # open the file 249 | ef = edf_open_file(filename.encode('utf-8'), 0, 1, 1, & errval) 250 | if errval < 0: 251 | raise IOError(f'Could not open: {filename}') 252 | e = edf_get_preamble_text(ef, buf, 1024) 253 | 254 | # initialize sample array 255 | num_elements = edf_get_element_count(ef) 256 | if ignore_samples: 257 | num_elements = 0 258 | cdef np.ndarray npsamples = np.ndarray((num_elements, 40), dtype=np.float64) 259 | cdef np.float64_t[:, :] samples = npsamples 260 | 261 | # parse samples and events 262 | trial = -1 263 | cnt = 0 264 | current_messages = {} 265 | current_event = {} 266 | message_accumulator = [] 267 | event_accumulator = [] 268 | while True: 269 | sample_type = edf_get_next_data(ef) 270 | if sample_type == NO_PENDING_ITEMS: 271 | edf_close_file(ef) 272 | break 273 | 274 | if not ignore_samples and (sample_type == SAMPLE_TYPE): 275 | fd = edf_get_float_data(ef) 276 | # Map fields explicitly into memory view 277 | samples[cnt, 0] = float(fd.fs.time) 278 | samples[cnt, 1] = float(fd.fs.px[0]) 279 | samples[cnt, 2] = float(fd.fs.px[1]) 280 | samples[cnt, 3] = float(fd.fs.py[0]) 281 | samples[cnt, 4] = float(fd.fs.py[1]) 282 | samples[cnt, 5] = float(fd.fs.hx[0]) 283 | samples[cnt, 6] = float(fd.fs.hx[1]) 284 | samples[cnt, 7] = float(fd.fs.hy[0]) 285 | samples[cnt, 8] = float(fd.fs.hy[1]) 286 | samples[cnt, 9] = float(fd.fs.pa[0]) 287 | samples[cnt, 10] = fd.fs.pa[1] 288 | samples[cnt, 11] = fd.fs.gx[0] 289 | samples[cnt, 12] = fd.fs.gx[1] 290 | samples[cnt, 13] = fd.fs.gy[0] 291 | samples[cnt, 14] = fd.fs.gy[1] 292 | samples[cnt, 15] = fd.fs.rx 293 | samples[cnt, 16] = fd.fs.ry 294 | 295 | samples[cnt, 17] = fd.fs.gxvel[0] 296 | samples[cnt, 18] = fd.fs.gxvel[1] 297 | samples[cnt, 19] = fd.fs.gyvel[0] 298 | samples[cnt, 20] = fd.fs.gyvel[1] 299 | samples[cnt, 21] = fd.fs.hxvel[0] 300 | samples[cnt, 22] = fd.fs.hxvel[1] 301 | samples[cnt, 23] = fd.fs.hyvel[0] 302 | samples[cnt, 24] = fd.fs.hyvel[1] 303 | samples[cnt, 25] = fd.fs.rxvel[0] 304 | samples[cnt, 26] = fd.fs.rxvel[1] 305 | samples[cnt, 27] = fd.fs.ryvel[0] 306 | samples[cnt, 28] = fd.fs.ryvel[1] 307 | 308 | samples[cnt, 29] = fd.fs.fgxvel[0] 309 | samples[cnt, 30] = fd.fs.fgyvel[0] 310 | samples[cnt, 31] = fd.fs.fhxvel[0] 311 | samples[cnt, 32] = fd.fs.fhyvel[0] 312 | samples[cnt, 33] = fd.fs.frxvel[0] 313 | samples[cnt, 34] = fd.fs.fryvel[0] 314 | 315 | # samples[cnt, 39:48] = fd.fs.hdata # head-tracker data 316 | # (not prescaled) 317 | samples[cnt, 35] = fd.fs.flags # flags to indicate contents 318 | samples[cnt, 36] = fd.fs.input # extra (input word) 319 | samples[cnt, 37] = fd.fs.buttons # button state & changes 320 | samples[cnt, 38] = fd.fs.htype # head-tracker data type 321 | samples[cnt, 39] = fd.fs.errors 322 | cnt += 1 323 | 324 | elif sample_type == MESSAGEEVENT: 325 | data = data2dict(sample_type, ef) 326 | trial = parse_message( 327 | data, trial, message_accumulator, message_filter, trial_marker 328 | ) 329 | 330 | else: 331 | data = data2dict(sample_type, ef) 332 | current_event = parse_datum( 333 | data, sample_type, trial, current_event, event_accumulator 334 | ) 335 | free(buf) 336 | # num_elements contained number of combined samples, messages, and events. This truncates 337 | # to only number of samples read (cnt). 338 | samples = samples[:cnt, :]; 339 | return samples, event_accumulator, message_accumulator 340 | -------------------------------------------------------------------------------- /src/pyedfread/parse.py: -------------------------------------------------------------------------------- 1 | from pyedfread import edf_read 2 | import numpy as np 3 | import pandas as pd 4 | import h5py 5 | import os 6 | import argparse 7 | 8 | 9 | def read_edf( 10 | filename, 11 | ignore_samples=False, 12 | message_filter=None, 13 | trial_marker="TRIALID", 14 | ): 15 | """ 16 | Parse an EDF file into a pandas.DataFrame. 17 | 18 | EDF files contain three types of data: samples, events and 19 | messages. Samples are what is recorded by the eye-tracker at each 20 | point in time. This contains for example instantaneous gaze 21 | position and pupil size. Events abstract from samples by defining 22 | fixations, saccades and blinks. Messages contain meta information 23 | about the recording such as user defined information and 24 | calibration information etc. 25 | 26 | Parameters 27 | ---------- 28 | filename : str 29 | Path to EDF file. 30 | 31 | ignore_samples : bool 32 | If true individual samples will not be saved, but only event 33 | averages. 34 | 35 | message_filter : list of str, optional 36 | Messages are kept only if they start with one of these strings. 37 | 38 | trial_marker : str, optional 39 | Messages that start with this string will be assumed to 40 | indicate the start of a trial. 41 | 42 | Returns 43 | ------- 44 | samples : pandas.DataFrame 45 | Sample information, including time, x, y, and pupil size. 46 | 47 | events : pandas.DataFrame 48 | Event information, including saccades and fixations. 49 | 50 | messages : pandas.DataFrame 51 | Message information. 52 | """ 53 | if not os.path.isfile(filename): 54 | raise RuntimeError(f"File does not exist: {filename}") 55 | 56 | samples, events, messages = edf_read.parse_edf( 57 | filename, ignore_samples, message_filter, trial_marker 58 | ) 59 | events = pd.DataFrame(events) 60 | messages = pd.DataFrame(messages) 61 | samples = pd.DataFrame(np.asarray(samples), columns=edf_read.sample_columns) 62 | return samples, events, messages 63 | 64 | 65 | def trials2events(events, messages): 66 | """Match trial meta information to individual events.""" 67 | return events.merge(messages, how="left", on=["trial"]) 68 | 69 | 70 | def save_human_understandable(samples, events, messages, path): 71 | """Save HDF with explicit mapping of string to numbers.""" 72 | f = h5py.File(path, "w") 73 | try: 74 | for name, data in zip( 75 | ["samples", "events", "messages"], [samples, events, messages] 76 | ): 77 | fm_group = f.create_group(name) 78 | for field in data.columns: 79 | try: 80 | fm_group.create_dataset( 81 | field, 82 | data=data[field], 83 | compression="gzip", 84 | compression_opts=1, 85 | shuffle=True, 86 | ) 87 | except TypeError: 88 | # Probably a string that can not be saved in hdf. 89 | # Map to numbers and save mapping in attrs. 90 | column = data[field].values.astype(str) 91 | mapping = dict((key, i) for i, key in enumerate(np.unique(column))) 92 | fm_group.create_dataset( 93 | field, data=np.array([mapping[val] for val in column]) 94 | ) 95 | fm_group.attrs[field + "mapping"] = str(mapping) 96 | finally: 97 | f.close() 98 | 99 | 100 | def convert_edf(): 101 | """Read an EDF file and write to an HDF file.""" 102 | parser = argparse.ArgumentParser() 103 | parser.add_argument("edffile", help="The EDF file you would like to parse") 104 | parser.add_argument("outputfile", help="Where to save the output") 105 | parser.add_argument( 106 | "-p", 107 | "--pandas_hdf", 108 | action="store_true", 109 | default=False, 110 | help="Use pandas to store HDF. Default simplifies HDF strucutre significantly (e.g. map strings to numbers and skip objects)", 111 | ) 112 | parser.add_argument( 113 | "-i", 114 | "--ignore-samples", 115 | action="store_true", 116 | default=False, 117 | help="Should the individual samples be stored in the output? Default is to read samples, ie. ignore-samples=False", 118 | ) 119 | parser.add_argument( 120 | "-j", 121 | "--join", 122 | action="store_true", 123 | default=False, 124 | help="If True events and messages will be joined into one big structure based on the trial number.", 125 | ) 126 | args = parser.parse_args() 127 | samples, events, messages = read_edf(args.edffile, ignore_samples=args.ignore_samples) 128 | 129 | if args.join: 130 | events = trials2events(events, messages) 131 | 132 | if not args.pandas_hdf: 133 | columns = [ 134 | col 135 | for col in messages.columns 136 | if not (messages[col].dtype == np.dtype("O")) 137 | ] 138 | messages = messages.loc[:, columns] 139 | save_human_understandable(samples, events, messages, args.outputfile) 140 | else: 141 | events.to_hdf(args.outputfile, "events", mode="w", complevel=9, complib="zlib") 142 | samples.to_hdf( 143 | args.outputfile, "samples", mode="w", complevel=9, complib="zlib" 144 | ) 145 | messages.to_hdf( 146 | args.outputfile, "messages", mode="w", complevel=9, complib="zlib" 147 | ) 148 | -------------------------------------------------------------------------------- /tests/nppBackup/test_read_edf.py.2025-04-15_115210.bak: -------------------------------------------------------------------------------- 1 | """ 2 | Test pyedfread by comparing against edf2asc output. 3 | 4 | Only compares readout of samples at the moment. 5 | """ 6 | 7 | from pkg_resources import resource_filename 8 | import pytest 9 | import numpy as np 10 | import pyedfread as edf 11 | 12 | 13 | @pytest.fixture(scope="session") 14 | def edf_data(): 15 | """Sample of read-in EDF data.""" 16 | edf_file = resource_filename("pyedfread", "data/SUB001.EDF") 17 | samples, events, messages = edf.read_edf(edf_file) 18 | data = {"samples": samples, "events": events, "messages": messages} 19 | return data 20 | 21 | 22 | def test_messages(edf_data): 23 | """Compare messages to known values.""" 24 | # have one extra sample compared to original and one extra message with 25 | # display coordinates 26 | messages = edf_data["messages"] 27 | expected = [ 28 | 'DISPLAY_COORDS 0 0 1919 1079', 29 | '', 30 | 'RECCFG CR 1000 2 1 R', 31 | 'ELCLCFG MTABLER', 32 | 'GAZE_COORDS 0.00 0.00 1919.00 1079.00', 33 | 'THRESHOLDS R 85 220', 34 | 'ELCL_WINDOW_SIZES 176 188 0 0', 35 | 'ELCL_PROC CENTROID (3)', 36 | 'ELCL_PCR_PARAM 5 3.0', 37 | '!MODE RECORD CR 1000 2 1 R' 38 | ] 39 | assert messages.loc[0:9, 'message'].to_list() == expected 40 | 41 | 42 | def test_events(edf_data): 43 | """Compare events to known values.""" 44 | events = edf_data["events"] 45 | assert events.shape == (500, 30) 46 | events = events.query("type!='blink'") 47 | assert events.shape == (485, 30) 48 | np.testing.assert_allclose(events["gavx"].mean(), 420.842680343156) 49 | np.testing.assert_allclose(events["gavy"].mean(), 205.318144115959) 50 | 51 | 52 | def test_validate_against_edf2asc(edf_data): 53 | """Compare EDF parsing to EDF2ASC output.""" 54 | samples = edf_data["samples"] 55 | samples = samples.loc[:, ("time", "gx_right", "gy_right", "pa_right")] 56 | samples = samples.round(1).set_index("time") 57 | 58 | # check the first n samples 59 | n = 100 60 | asc_file = resource_filename("pyedfread", "data/SUB001.asc") 61 | with open(asc_file) as asc: 62 | for i, line in enumerate(asc): 63 | if i == n: 64 | break 65 | x = ( 66 | line.strip() 67 | .replace("...", "") 68 | .replace("\t", "") 69 | .replace(" ", " ") 70 | .split(" ") 71 | ) 72 | if len(x) == 4: 73 | try: 74 | # SAMPLE, X, Y, PA 75 | x = [float(n) for n in x] 76 | row = samples.loc[x[0], :] 77 | np.testing.assert_allclose(x[1:], row.to_numpy()) 78 | except ValueError: 79 | pass 80 | 81 | 82 | def test_ignore_samples(): 83 | """Test option to ignore samples in EDF file.""" 84 | edf_file = resource_filename('pyedfread', 'data/SUB001.EDF') 85 | samples, events, messages = edf.read_edf(edf_file, ignore_samples=True) 86 | assert samples.shape[0] == 0 87 | -------------------------------------------------------------------------------- /tests/nppBackup/test_read_edf.py.2025-04-15_115222.bak: -------------------------------------------------------------------------------- 1 | """ 2 | Test pyedfread by comparing against edf2asc output. 3 | 4 | Only compares readout of samples at the moment. 5 | """ 6 | 7 | from pkg_resources import resource_filename 8 | import pytest 9 | import numpy as np 10 | import pyedfread as edf 11 | 12 | 13 | @pytest.fixture(scope="session") 14 | def edf_data(): 15 | """Sample of read-in EDF data.""" 16 | edf_file = resource_filename("pyedfread", "data/SUB001.EDF") 17 | samples, events, messages = edf.read_edf(edf_file) 18 | data = {"samples": samples, "events": events, "messages": messages} 19 | return data 20 | 21 | def test_samples(edf_data): 22 | samples = edf_data["samples"] 23 | 24 | assert samples[samples.time.diff()!=1].shape == (1,40) 25 | 26 | 27 | def test_messages(edf_data): 28 | """Compare messages to known values.""" 29 | # have one extra sample compared to original and one extra message with 30 | # display coordinates 31 | messages = edf_data["messages"] 32 | expected = [ 33 | 'DISPLAY_COORDS 0 0 1919 1079', 34 | '', 35 | 'RECCFG CR 1000 2 1 R', 36 | 'ELCLCFG MTABLER', 37 | 'GAZE_COORDS 0.00 0.00 1919.00 1079.00', 38 | 'THRESHOLDS R 85 220', 39 | 'ELCL_WINDOW_SIZES 176 188 0 0', 40 | 'ELCL_PROC CENTROID (3)', 41 | 'ELCL_PCR_PARAM 5 3.0', 42 | '!MODE RECORD CR 1000 2 1 R' 43 | ] 44 | assert messages.loc[0:9, 'message'].to_list() == expected 45 | 46 | 47 | def test_events(edf_data): 48 | """Compare events to known values.""" 49 | events = edf_data["events"] 50 | assert events.shape == (500, 30) 51 | events = events.query("type!='blink'") 52 | assert events.shape == (485, 30) 53 | np.testing.assert_allclose(events["gavx"].mean(), 420.842680343156) 54 | np.testing.assert_allclose(events["gavy"].mean(), 205.318144115959) 55 | 56 | 57 | def test_validate_against_edf2asc(edf_data): 58 | """Compare EDF parsing to EDF2ASC output.""" 59 | samples = edf_data["samples"] 60 | samples = samples.loc[:, ("time", "gx_right", "gy_right", "pa_right")] 61 | samples = samples.round(1).set_index("time") 62 | 63 | # check the first n samples 64 | n = 100 65 | asc_file = resource_filename("pyedfread", "data/SUB001.asc") 66 | with open(asc_file) as asc: 67 | for i, line in enumerate(asc): 68 | if i == n: 69 | break 70 | x = ( 71 | line.strip() 72 | .replace("...", "") 73 | .replace("\t", "") 74 | .replace(" ", " ") 75 | .split(" ") 76 | ) 77 | if len(x) == 4: 78 | try: 79 | # SAMPLE, X, Y, PA 80 | x = [float(n) for n in x] 81 | row = samples.loc[x[0], :] 82 | np.testing.assert_allclose(x[1:], row.to_numpy()) 83 | except ValueError: 84 | pass 85 | 86 | 87 | def test_ignore_samples(): 88 | """Test option to ignore samples in EDF file.""" 89 | edf_file = resource_filename('pyedfread', 'data/SUB001.EDF') 90 | samples, events, messages = edf.read_edf(edf_file, ignore_samples=True) 91 | assert samples.shape[0] == 0 92 | -------------------------------------------------------------------------------- /tests/test_read_edf.py: -------------------------------------------------------------------------------- 1 | """ 2 | Test pyedfread by comparing against edf2asc output. 3 | 4 | Only compares readout of samples at the moment. 5 | """ 6 | 7 | from pkg_resources import resource_filename 8 | import pytest 9 | import numpy as np 10 | import pyedfread as edf 11 | 12 | 13 | @pytest.fixture(scope="session") 14 | def edf_data(): 15 | """Sample of read-in EDF data.""" 16 | edf_file = resource_filename("pyedfread", "data/SUB001.EDF") 17 | samples, events, messages = edf.read_edf(edf_file) 18 | data = {"samples": samples, "events": events, "messages": messages} 19 | return data 20 | 21 | def test_samples(edf_data): 22 | samples = edf_data["samples"] 23 | 24 | # test for #37 25 | assert samples[samples.time.diff()!=1].shape == (1,40) 26 | 27 | 28 | def test_messages(edf_data): 29 | """Compare messages to known values.""" 30 | # have one extra sample compared to original and one extra message with 31 | # display coordinates 32 | messages = edf_data["messages"] 33 | expected = [ 34 | 'DISPLAY_COORDS 0 0 1919 1079', 35 | '', 36 | 'RECCFG CR 1000 2 1 R', 37 | 'ELCLCFG MTABLER', 38 | 'GAZE_COORDS 0.00 0.00 1919.00 1079.00', 39 | 'THRESHOLDS R 85 220', 40 | 'ELCL_WINDOW_SIZES 176 188 0 0', 41 | 'ELCL_PROC CENTROID (3)', 42 | 'ELCL_PCR_PARAM 5 3.0', 43 | '!MODE RECORD CR 1000 2 1 R' 44 | ] 45 | assert messages.loc[0:9, 'message'].to_list() == expected 46 | 47 | 48 | def test_events(edf_data): 49 | """Compare events to known values.""" 50 | events = edf_data["events"] 51 | assert events.shape == (500, 30) 52 | events = events.query("type!='blink'") 53 | assert events.shape == (485, 30) 54 | np.testing.assert_allclose(events["gavx"].mean(), 420.842680343156) 55 | np.testing.assert_allclose(events["gavy"].mean(), 205.318144115959) 56 | 57 | 58 | def test_validate_against_edf2asc(edf_data): 59 | """Compare EDF parsing to EDF2ASC output.""" 60 | samples = edf_data["samples"] 61 | samples = samples.loc[:, ("time", "gx_right", "gy_right", "pa_right")] 62 | samples = samples.round(1).set_index("time") 63 | 64 | # check the first n samples 65 | n = 100 66 | asc_file = resource_filename("pyedfread", "data/SUB001.asc") 67 | with open(asc_file) as asc: 68 | for i, line in enumerate(asc): 69 | if i == n: 70 | break 71 | x = ( 72 | line.strip() 73 | .replace("...", "") 74 | .replace("\t", "") 75 | .replace(" ", " ") 76 | .split(" ") 77 | ) 78 | if len(x) == 4: 79 | try: 80 | # SAMPLE, X, Y, PA 81 | x = [float(n) for n in x] 82 | row = samples.loc[x[0], :] 83 | np.testing.assert_allclose(x[1:], row.to_numpy()) 84 | except ValueError: 85 | pass 86 | 87 | 88 | def test_ignore_samples(): 89 | """Test option to ignore samples in EDF file.""" 90 | edf_file = resource_filename('pyedfread', 'data/SUB001.EDF') 91 | samples, events, messages = edf.read_edf(edf_file, ignore_samples=True) 92 | assert samples.shape[0] == 0 93 | -------------------------------------------------------------------------------- /tests/test_read_sections.py: -------------------------------------------------------------------------------- 1 | """Test reading from different sections of EDF data.""" 2 | 3 | from pkg_resources import resource_filename 4 | import pytest 5 | import pyedfread as edf 6 | 7 | 8 | @pytest.fixture(scope="session") 9 | def edf_file(): 10 | filepath = resource_filename("pyedfread", "data/SUB001.EDF") 11 | return filepath 12 | 13 | 14 | def test_read_preamble(edf_file): 15 | """Test reading the preamble section.""" 16 | preamble = edf.read_preamble(edf_file) 17 | expected = """** DATE: Tue Jun 27 02:04:46 2017 18 | ** TYPE: EDF_FILE BINARY EVENT SAMPLE TAGGED 19 | ** VERSION: EYELINK II 1 20 | ** SOURCE: EYELINK CL 21 | ** EYELINK II CL v5.04 Sep 25 2014 22 | ** CAMERA: Eyelink GL Version 1.2 Sensor=AI7 23 | ** SERIAL NUMBER: CLG-BBF30 24 | ** CAMERA_CONFIG: BBF30200.SCD 25 | """ 26 | assert preamble == expected 27 | 28 | 29 | def test_read_messages(edf_file): 30 | """Read reading messages.""" 31 | messages = edf.read_messages(edf_file) 32 | expected = [ 33 | 'DISPLAY_COORDS 0 0 1919 1079', 34 | '', 35 | 'RECCFG CR 1000 2 1 R', 36 | 'ELCLCFG MTABLER', 37 | 'GAZE_COORDS 0.00 0.00 1919.00 1079.00', 38 | 'THRESHOLDS R 85 220', 39 | 'ELCL_WINDOW_SIZES 176 188 0 0', 40 | 'ELCL_PROC CENTROID (3)', 41 | 'ELCL_PCR_PARAM 5 3.0', 42 | 'ELCL_PCR_PARAM 5 3.0', 43 | '!MODE RECORD CR 1000 2 1 R' 44 | ] 45 | assert messages[:len(expected)] == expected 46 | -------------------------------------------------------------------------------- /tox.ini: -------------------------------------------------------------------------------- 1 | [tox] 2 | env_list = 3 | py310 4 | minversion = 4.15.1 5 | 6 | [testenv] 7 | description = run the tests with pytest 8 | package = wheel 9 | wheel_build_env = .pkg 10 | deps = 11 | pytest>=6 12 | commands = 13 | pytest {tty:--color=yes} {posargs} 14 | --------------------------------------------------------------------------------