├── .gitignore ├── LICENSE.md ├── addon ├── globalPlugins │ └── audioScreen │ │ ├── __init__.py │ │ ├── imagePlayer.py │ │ └── screenBitmap.py └── manifest.ini └── readme.md /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyc 2 | *.pyo 3 | *.nvda-addon 4 | addon/globalPlugins/audioScreen/deps 5 | -------------------------------------------------------------------------------- /LICENSE.md: -------------------------------------------------------------------------------- 1 | GNU GENERAL PUBLIC LICENSE 2 | Version 2, June 1991 3 | 4 | Copyright (C) 1989, 1991 Free Software Foundation, Inc., 5 | 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA 6 | Everyone is permitted to copy and distribute verbatim copies 7 | of this license document, but changing it is not allowed. 8 | 9 | Preamble 10 | 11 | The licenses for most software are designed to take away your 12 | freedom to share and change it. By contrast, the GNU General Public 13 | License is intended to guarantee your freedom to share and change free 14 | software--to make sure the software is free for all its users. This 15 | General Public License applies to most of the Free Software 16 | Foundation's software and to any other program whose authors commit to 17 | using it. (Some other Free Software Foundation software is covered by 18 | the GNU Lesser General Public License instead.) You can apply it to 19 | your programs, too. 20 | 21 | When we speak of free software, we are referring to freedom, not 22 | price. Our General Public Licenses are designed to make sure that you 23 | have the freedom to distribute copies of free software (and charge for 24 | this service if you wish), that you receive source code or can get it 25 | if you want it, that you can change the software or use pieces of it 26 | in new free programs; and that you know you can do these things. 27 | 28 | To protect your rights, we need to make restrictions that forbid 29 | anyone to deny you these rights or to ask you to surrender the rights. 30 | These restrictions translate to certain responsibilities for you if you 31 | distribute copies of the software, or if you modify it. 32 | 33 | For example, if you distribute copies of such a program, whether 34 | gratis or for a fee, you must give the recipients all the rights that 35 | you have. You must make sure that they, too, receive or can get the 36 | source code. And you must show them these terms so they know their 37 | rights. 38 | 39 | We protect your rights with two steps: (1) copyright the software, and 40 | (2) offer you this license which gives you legal permission to copy, 41 | distribute and/or modify the software. 42 | 43 | Also, for each author's protection and ours, we want to make certain 44 | that everyone understands that there is no warranty for this free 45 | software. If the software is modified by someone else and passed on, we 46 | want its recipients to know that what they have is not the original, so 47 | that any problems introduced by others will not reflect on the original 48 | authors' reputations. 49 | 50 | Finally, any free program is threatened constantly by software 51 | patents. We wish to avoid the danger that redistributors of a free 52 | program will individually obtain patent licenses, in effect making the 53 | program proprietary. To prevent this, we have made it clear that any 54 | patent must be licensed for everyone's free use or not licensed at all. 55 | 56 | The precise terms and conditions for copying, distribution and 57 | modification follow. 58 | 59 | GNU GENERAL PUBLIC LICENSE 60 | TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION 61 | 62 | 0. This License applies to any program or other work which contains 63 | a notice placed by the copyright holder saying it may be distributed 64 | under the terms of this General Public License. The "Program", below, 65 | refers to any such program or work, and a "work based on the Program" 66 | means either the Program or any derivative work under copyright law: 67 | that is to say, a work containing the Program or a portion of it, 68 | either verbatim or with modifications and/or translated into another 69 | language. (Hereinafter, translation is included without limitation in 70 | the term "modification".) Each licensee is addressed as "you". 71 | 72 | Activities other than copying, distribution and modification are not 73 | covered by this License; they are outside its scope. The act of 74 | running the Program is not restricted, and the output from the Program 75 | is covered only if its contents constitute a work based on the 76 | Program (independent of having been made by running the Program). 77 | Whether that is true depends on what the Program does. 78 | 79 | 1. You may copy and distribute verbatim copies of the Program's 80 | source code as you receive it, in any medium, provided that you 81 | conspicuously and appropriately publish on each copy an appropriate 82 | copyright notice and disclaimer of warranty; keep intact all the 83 | notices that refer to this License and to the absence of any warranty; 84 | and give any other recipients of the Program a copy of this License 85 | along with the Program. 86 | 87 | You may charge a fee for the physical act of transferring a copy, and 88 | you may at your option offer warranty protection in exchange for a fee. 89 | 90 | 2. You may modify your copy or copies of the Program or any portion 91 | of it, thus forming a work based on the Program, and copy and 92 | distribute such modifications or work under the terms of Section 1 93 | above, provided that you also meet all of these conditions: 94 | 95 | a) You must cause the modified files to carry prominent notices 96 | stating that you changed the files and the date of any change. 97 | 98 | b) You must cause any work that you distribute or publish, that in 99 | whole or in part contains or is derived from the Program or any 100 | part thereof, to be licensed as a whole at no charge to all third 101 | parties under the terms of this License. 102 | 103 | c) If the modified program normally reads commands interactively 104 | when run, you must cause it, when started running for such 105 | interactive use in the most ordinary way, to print or display an 106 | announcement including an appropriate copyright notice and a 107 | notice that there is no warranty (or else, saying that you provide 108 | a warranty) and that users may redistribute the program under 109 | these conditions, and telling the user how to view a copy of this 110 | License. (Exception: if the Program itself is interactive but 111 | does not normally print such an announcement, your work based on 112 | the Program is not required to print an announcement.) 113 | 114 | These requirements apply to the modified work as a whole. If 115 | identifiable sections of that work are not derived from the Program, 116 | and can be reasonably considered independent and separate works in 117 | themselves, then this License, and its terms, do not apply to those 118 | sections when you distribute them as separate works. But when you 119 | distribute the same sections as part of a whole which is a work based 120 | on the Program, the distribution of the whole must be on the terms of 121 | this License, whose permissions for other licensees extend to the 122 | entire whole, and thus to each and every part regardless of who wrote it. 123 | 124 | Thus, it is not the intent of this section to claim rights or contest 125 | your rights to work written entirely by you; rather, the intent is to 126 | exercise the right to control the distribution of derivative or 127 | collective works based on the Program. 128 | 129 | In addition, mere aggregation of another work not based on the Program 130 | with the Program (or with a work based on the Program) on a volume of 131 | a storage or distribution medium does not bring the other work under 132 | the scope of this License. 133 | 134 | 3. You may copy and distribute the Program (or a work based on it, 135 | under Section 2) in object code or executable form under the terms of 136 | Sections 1 and 2 above provided that you also do one of the following: 137 | 138 | a) Accompany it with the complete corresponding machine-readable 139 | source code, which must be distributed under the terms of Sections 140 | 1 and 2 above on a medium customarily used for software interchange; or, 141 | 142 | b) Accompany it with a written offer, valid for at least three 143 | years, to give any third party, for a charge no more than your 144 | cost of physically performing source distribution, a complete 145 | machine-readable copy of the corresponding source code, to be 146 | distributed under the terms of Sections 1 and 2 above on a medium 147 | customarily used for software interchange; or, 148 | 149 | c) Accompany it with the information you received as to the offer 150 | to distribute corresponding source code. (This alternative is 151 | allowed only for noncommercial distribution and only if you 152 | received the program in object code or executable form with such 153 | an offer, in accord with Subsection b above.) 154 | 155 | The source code for a work means the preferred form of the work for 156 | making modifications to it. For an executable work, complete source 157 | code means all the source code for all modules it contains, plus any 158 | associated interface definition files, plus the scripts used to 159 | control compilation and installation of the executable. However, as a 160 | special exception, the source code distributed need not include 161 | anything that is normally distributed (in either source or binary 162 | form) with the major components (compiler, kernel, and so on) of the 163 | operating system on which the executable runs, unless that component 164 | itself accompanies the executable. 165 | 166 | If distribution of executable or object code is made by offering 167 | access to copy from a designated place, then offering equivalent 168 | access to copy the source code from the same place counts as 169 | distribution of the source code, even though third parties are not 170 | compelled to copy the source along with the object code. 171 | 172 | 4. You may not copy, modify, sublicense, or distribute the Program 173 | except as expressly provided under this License. Any attempt 174 | otherwise to copy, modify, sublicense or distribute the Program is 175 | void, and will automatically terminate your rights under this License. 176 | However, parties who have received copies, or rights, from you under 177 | this License will not have their licenses terminated so long as such 178 | parties remain in full compliance. 179 | 180 | 5. You are not required to accept this License, since you have not 181 | signed it. However, nothing else grants you permission to modify or 182 | distribute the Program or its derivative works. These actions are 183 | prohibited by law if you do not accept this License. Therefore, by 184 | modifying or distributing the Program (or any work based on the 185 | Program), you indicate your acceptance of this License to do so, and 186 | all its terms and conditions for copying, distributing or modifying 187 | the Program or works based on it. 188 | 189 | 6. Each time you redistribute the Program (or any work based on the 190 | Program), the recipient automatically receives a license from the 191 | original licensor to copy, distribute or modify the Program subject to 192 | these terms and conditions. You may not impose any further 193 | restrictions on the recipients' exercise of the rights granted herein. 194 | You are not responsible for enforcing compliance by third parties to 195 | this License. 196 | 197 | 7. If, as a consequence of a court judgment or allegation of patent 198 | infringement or for any other reason (not limited to patent issues), 199 | conditions are imposed on you (whether by court order, agreement or 200 | otherwise) that contradict the conditions of this License, they do not 201 | excuse you from the conditions of this License. If you cannot 202 | distribute so as to satisfy simultaneously your obligations under this 203 | License and any other pertinent obligations, then as a consequence you 204 | may not distribute the Program at all. For example, if a patent 205 | license would not permit royalty-free redistribution of the Program by 206 | all those who receive copies directly or indirectly through you, then 207 | the only way you could satisfy both it and this License would be to 208 | refrain entirely from distribution of the Program. 209 | 210 | If any portion of this section is held invalid or unenforceable under 211 | any particular circumstance, the balance of the section is intended to 212 | apply and the section as a whole is intended to apply in other 213 | circumstances. 214 | 215 | It is not the purpose of this section to induce you to infringe any 216 | patents or other property right claims or to contest validity of any 217 | such claims; this section has the sole purpose of protecting the 218 | integrity of the free software distribution system, which is 219 | implemented by public license practices. Many people have made 220 | generous contributions to the wide range of software distributed 221 | through that system in reliance on consistent application of that 222 | system; it is up to the author/donor to decide if he or she is willing 223 | to distribute software through any other system and a licensee cannot 224 | impose that choice. 225 | 226 | This section is intended to make thoroughly clear what is believed to 227 | be a consequence of the rest of this License. 228 | 229 | 8. If the distribution and/or use of the Program is restricted in 230 | certain countries either by patents or by copyrighted interfaces, the 231 | original copyright holder who places the Program under this License 232 | may add an explicit geographical distribution limitation excluding 233 | those countries, so that distribution is permitted only in or among 234 | countries not thus excluded. In such case, this License incorporates 235 | the limitation as if written in the body of this License. 236 | 237 | 9. The Free Software Foundation may publish revised and/or new versions 238 | of the General Public License from time to time. Such new versions will 239 | be similar in spirit to the present version, but may differ in detail to 240 | address new problems or concerns. 241 | 242 | Each version is given a distinguishing version number. If the Program 243 | specifies a version number of this License which applies to it and "any 244 | later version", you have the option of following the terms and conditions 245 | either of that version or of any later version published by the Free 246 | Software Foundation. If the Program does not specify a version number of 247 | this License, you may choose any version ever published by the Free Software 248 | Foundation. 249 | 250 | 10. If you wish to incorporate parts of the Program into other free 251 | programs whose distribution conditions are different, write to the author 252 | to ask for permission. For software which is copyrighted by the Free 253 | Software Foundation, write to the Free Software Foundation; we sometimes 254 | make exceptions for this. Our decision will be guided by the two goals 255 | of preserving the free status of all derivatives of our free software and 256 | of promoting the sharing and reuse of software generally. 257 | 258 | NO WARRANTY 259 | 260 | 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY 261 | FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN 262 | OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES 263 | PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED 264 | OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF 265 | MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS 266 | TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE 267 | PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, 268 | REPAIR OR CORRECTION. 269 | 270 | 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING 271 | WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR 272 | REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, 273 | INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING 274 | OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED 275 | TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY 276 | YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER 277 | PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE 278 | POSSIBILITY OF SUCH DAMAGES. 279 | 280 | END OF TERMS AND CONDITIONS 281 | 282 | How to Apply These Terms to Your New Programs 283 | 284 | If you develop a new program, and you want it to be of the greatest 285 | possible use to the public, the best way to achieve this is to make it 286 | free software which everyone can redistribute and change under these terms. 287 | 288 | To do so, attach the following notices to the program. It is safest 289 | to attach them to the start of each source file to most effectively 290 | convey the exclusion of warranty; and each file should have at least 291 | the "copyright" line and a pointer to where the full notice is found. 292 | 293 | {description} 294 | Copyright (C) {year} {fullname} 295 | 296 | This program is free software; you can redistribute it and/or modify 297 | it under the terms of the GNU General Public License as published by 298 | the Free Software Foundation; either version 2 of the License, or 299 | (at your option) any later version. 300 | 301 | This program is distributed in the hope that it will be useful, 302 | but WITHOUT ANY WARRANTY; without even the implied warranty of 303 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 304 | GNU General Public License for more details. 305 | 306 | You should have received a copy of the GNU General Public License along 307 | with this program; if not, write to the Free Software Foundation, Inc., 308 | 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. 309 | 310 | Also add information on how to contact you by electronic and paper mail. 311 | 312 | If the program is interactive, make it output a short notice like this 313 | when it starts in an interactive mode: 314 | 315 | Gnomovision version 69, Copyright (C) year name of author 316 | Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. 317 | This is free software, and you are welcome to redistribute it 318 | under certain conditions; type `show c' for details. 319 | 320 | The hypothetical commands `show w' and `show c' should show the appropriate 321 | parts of the General Public License. Of course, the commands you use may 322 | be called something other than `show w' and `show c'; they could even be 323 | mouse-clicks or menu items--whatever suits your program. 324 | 325 | You should also get your employer (if you work as a programmer) or your 326 | school, if any, to sign a "copyright disclaimer" for the program, if 327 | necessary. Here is a sample; alter the names: 328 | 329 | Yoyodyne, Inc., hereby disclaims all copyright interest in the program 330 | `Gnomovision' (which makes passes at compilers) written by James Hacker. 331 | 332 | {signature of Ty Coon}, 1 April 1989 333 | Ty Coon, President of Vice 334 | 335 | This General Public License does not permit incorporating your program into 336 | proprietary programs. If your program is a subroutine library, you may 337 | consider it more useful to permit linking proprietary applications with the 338 | library. If this is what you want to do, use the GNU Lesser General 339 | Public License instead of this License. 340 | -------------------------------------------------------------------------------- /addon/globalPlugins/audioScreen/__init__.py: -------------------------------------------------------------------------------- 1 | import sys 2 | import os 3 | 4 | sys.path.append(os.path.join(os.path.dirname(__file__),'deps')) 5 | 6 | import ctypes 7 | import wx 8 | from . import libaudioverse 9 | import config 10 | from gui.settingsDialogs import SettingsDialog 11 | import gui 12 | import globalPluginHandler 13 | import touchHandler 14 | import globalCommands 15 | import api 16 | from . import screenBitmap 17 | import textInfos 18 | import ui 19 | from . import imagePlayer 20 | 21 | class AudioScreenDialog(SettingsDialog): 22 | title=_("AudioScreen settings") 23 | 24 | def __init__(self,parent,plugin): 25 | self.plugin=plugin 26 | super(AudioScreenDialog,self).__init__(parent) 27 | 28 | def makeSettings(self,settingsSizer): 29 | generalSizer=wx.StaticBoxSizer(wx.StaticBox(self,wx.ID_ANY,_("General")),wx.VERTICAL) 30 | modeChoiceSizer=wx.BoxSizer(wx.HORIZONTAL) 31 | modeChoiceSizer.Add(wx.StaticText(self,wx.ID_ANY,_("Mode"))) 32 | self.modeChoice=wx.Choice(self,wx.ID_ANY,choices=[x[0] for x in self.plugin.audioScreenModes]) 33 | self.modeChoice.SetSelection(self.plugin.curAudioScreenMode) 34 | modeChoiceSizer.Add(self.modeChoice) 35 | generalSizer.Add(modeChoiceSizer) 36 | settingsSizer.Add(generalSizer) 37 | modesSizer=wx.BoxSizer(wx.HORIZONTAL) 38 | self.modeControls=[] 39 | for mode in self.plugin.audioScreenModes[1:]: 40 | modeSizer=wx.StaticBoxSizer(wx.StaticBox(self,wx.ID_ANY,mode[0]),wx.VERTICAL) 41 | modeConf=config.conf["audioScreen_%s"%mode[1].__name__] 42 | for v in mode[2]: 43 | if v[1]=='boolean': 44 | control=wx.CheckBox(self,wx.ID_ANY,label=v[3]) 45 | control.SetValue(modeConf[v[0]]) 46 | modeSizer.Add(control) 47 | else: 48 | fieldSizer=wx.BoxSizer(wx.HORIZONTAL) 49 | fieldSizer.Add(wx.StaticText(self,wx.ID_ANY,v[3])) 50 | control=wx.TextCtrl(self,wx.ID_ANY) 51 | control.SetValue(str(modeConf[v[0]])) 52 | fieldSizer.Add(control) 53 | modeSizer.Add(fieldSizer) 54 | self.modeControls.append(control) 55 | modesSizer.Add(modeSizer) 56 | settingsSizer.Add(modesSizer) 57 | 58 | def postInit(self): 59 | self.modeChoice.SetFocus() 60 | 61 | def onOk(self,evt): 62 | modeControlIndex=0 63 | for mode in GlobalPlugin.audioScreenModes[1:]: 64 | modeConf=config.conf["audioScreen_%s"%mode[1].__name__] 65 | for v in mode[2]: 66 | control=self.modeControls[modeControlIndex] 67 | if v[1]=='boolean': 68 | modeConf[v[0]]=control.IsChecked() 69 | else: 70 | try: 71 | value=float(control.Value) if v[1]=='float' else int(control.Value) 72 | except: 73 | value=v[2] 74 | modeConf[v[0]]=value 75 | modeControlIndex+=1 76 | curMode=self.modeChoice.GetSelection() 77 | self.plugin.setMode(curMode) 78 | super(AudioScreenDialog,self).onOk(evt) 79 | 80 | class GlobalPlugin(globalPluginHandler.GlobalPlugin): 81 | 82 | audioScreenModes=[ 83 | (_("Off"),None), 84 | (_("pitch stereo grey"),imagePlayer.ImagePlayer_pitchStereoGrey,[ 85 | ("reverseBrightness","boolean",False,_("Reverse brightness (useful for white on black)")), 86 | ("width","integer",176,_("Number of columns in stereo field")), 87 | ("height","integer",64,_("Number of rows (frequencies)")), 88 | ("lowFreq","float",500.0,_("Lowest frequency in HZ")), 89 | ("highFreq","float",5000.0,_("highest frequency in HZ")), 90 | ("sweepDelay","float",0.5,_("Initial stereo sweep Delay in seconds")), 91 | ("sweepDuration","float",4.0,_("Duration of stereo audio sweep in seconds")), 92 | ("sweepCount","integer",4,_("Numver of stereo sweeps")), 93 | ("captureWidth","integer",32,_("width (in pixels) of the rectangle at the point under your finger / the mouse")), 94 | ("captureHeight","integer",32,_("height (in pixels) of the rectangle at the point under your finger / the mouse")), 95 | ]), 96 | (_("HSV Color"),imagePlayer.ImagePlayer_hsv,[ 97 | ("width","integer",2,_("Horizontal length of capture area in pixels")), 98 | ("height","integer",2,_("Vertical length of capture area in pixels")), 99 | ("lowFreq","float",90.0,_("Lowest frequency (blue) in HZ")), 100 | ("highFreq","float",5760,_("highest frequency (red) in HZ")), 101 | ]), 102 | ] 103 | for mode in audioScreenModes[1:]: 104 | config.conf.spec["audioScreen_%s"%mode[1].__name__]={v[0]:"%s(default=%s)"%v[1:3] for v in mode[2]} 105 | 106 | def __init__(self): 107 | libaudioverse.initialize() 108 | self._lastRect=None 109 | self.curAudioScreenMode=0 110 | self.imagePlayer=self.screenBitmap=None 111 | item = gui.mainFrame.sysTrayIcon.preferencesMenu.Append(wx.ID_ANY,_("&AudioScreen..."),_("AudioScreen")) 112 | gui.mainFrame.sysTrayIcon.Bind(wx.EVT_MENU, self.script_showUI, item) 113 | super(GlobalPlugin,self).__init__() 114 | 115 | def terminate(self): 116 | libaudioverse.shutdown() 117 | 118 | def playPoint(self,x,y): 119 | if not self.imagePlayer: return 120 | screenWidth,screenHeight=api.getDesktopObject().location[2:] 121 | width=self.captureWidth 122 | height=self.captureHeight 123 | x=x-(width/2) 124 | y=y-(height/2) 125 | self.playRect(x,y,width,height) 126 | 127 | def playRect(self,x,y,width,height,detailed=False,forceRestart=False): 128 | if not self.imagePlayer: return 129 | rect=(x,y,width,height) 130 | if not forceRestart and rect==self._lastRect: 131 | return 132 | self._lastRect=rect 133 | buffer=self.screenBitmap.captureImage(x,y,width,height) 134 | self.imagePlayer.setNewImage(buffer,detailed=detailed) 135 | 136 | def stopPlaying(self): 137 | if self.imagePlayer: self.imagePlayer.setNewImage(None) 138 | 139 | def event_mouseMove(self,obj,nextHandler,x=None,y=None): 140 | nextHandler() 141 | if touchHandler.handler: return 142 | self.playPoint(x,y) 143 | 144 | def setMode(self,modeID,report=False): 145 | self.curAudioScreenMode=modeID 146 | modeInfo=self.audioScreenModes[modeID] 147 | if self.imagePlayer: 148 | imagePlayer=self.imagePlayer 149 | self.imagePlayer=None 150 | imagePlayer.terminate() 151 | self.screenBitmap=None 152 | if modeInfo[1] is None: 153 | if report: ui.message(_("AudioScreen off")) 154 | else: 155 | modeConf={k:v for k,v in config.conf["audioScreen_%s"%modeInfo[1].__name__].items()} 156 | self.captureWidth=modeConf.pop('captureWidth',modeConf['width']) 157 | self.captureHeight=modeConf.pop('captureHeight',modeConf['height']) 158 | self.imagePlayer=modeInfo[1](**modeConf) 159 | self.screenBitmap=screenBitmap.ScreenBitmap(self.imagePlayer.width,self.imagePlayer.height) 160 | if report: 161 | inputType=_("touch input") if touchHandler.handler else _("mouse input") 162 | ui.message(_("AudioScreen mode {mode}, {inputType}").format(mode=modeInfo[0],inputType=inputType)) 163 | 164 | def script_toggleAudioScreen(self,gesture): 165 | self.setMode((self.curAudioScreenMode+1)%len(self.audioScreenModes),report=True) 166 | script_toggleAudioScreen.__doc__="Toggles AudioScreen between several modes" 167 | 168 | def script_toggleBrightness(self,gesture): 169 | if not self.imagePlayer: 170 | ui.message(_("Audio screen currently off")) 171 | return 172 | rb=not self.imagePlayer.reverseBrightness 173 | if not rb: 174 | ui.message("Dark on light") 175 | else: 176 | ui.message("Light on dark") 177 | self.imagePlayer.reverseBrightness=rb 178 | script_toggleBrightness.__doc__="Toggles between light on dark, and dark on light" 179 | 180 | def script_hover(self,gesture): 181 | preheldTracker=getattr(gesture,'preheldTracker',None) 182 | if preheldTracker: 183 | xList=[tracker.x for tracker in preheldTracker.childTrackers] 184 | xList.append(preheldTracker.x) 185 | xList.append(gesture.tracker.x) 186 | yList=[tracker.y for tracker in preheldTracker.childTrackers] 187 | yList.append(preheldTracker.y) 188 | yList.append(gesture.tracker.y) 189 | minX=min(xList) 190 | maxX=max(xList) 191 | minY=min(yList) 192 | maxY=max(yList) 193 | self.playRect(minX,minY,maxX-minX,maxY-minY,detailed=True) 194 | else: 195 | self.playPoint(gesture.tracker.x,gesture.tracker.y) 196 | script=globalCommands.commands.getScript(gesture) 197 | if script: script(gesture) 198 | script_hover.__doc__=_("Plays the image under your fingers") 199 | 200 | def script_hoverUp(self,gesture): 201 | self.stopPlaying() 202 | script=globalCommands.commands.getScript(gesture) 203 | if script: script(gesture) 204 | script_hoverUp.__doc__=_("Stops audioScreen playback") 205 | 206 | def script_playNavigatorObject(self,gesture): 207 | if not self.imagePlayer: 208 | ui.message(_("AudioScreen disabled")) 209 | return 210 | obj=api.getNavigatorObject() 211 | x,y,w,h=obj.location 212 | self.playRect(x,y,w,h,detailed=True,forceRestart=True) 213 | script_playNavigatorObject.__doc__=_("Plays the image of the current navigator object") 214 | 215 | def script_showUI(self,gesture): 216 | wx.CallAfter(gui.mainFrame._popupSettingsDialog,AudioScreenDialog,self) 217 | 218 | __gestures={ 219 | "ts:hoverDown":"hover", 220 | "ts:hold+hoverDown":"hover", 221 | "ts:hover":"hover", 222 | "ts:hold+hover":"hover", 223 | "ts:hoverUp":"hoverUp", 224 | "ts:hold+hoverUp":"hover", 225 | "kb:NVDA+Shift+a":"showUI", 226 | "kb:NVDA+Control+a":"toggleAudioScreen", 227 | "kb:alt+NVDA+a":"playNavigatorObject", 228 | } 229 | 230 | -------------------------------------------------------------------------------- /addon/globalPlugins/audioScreen/imagePlayer.py: -------------------------------------------------------------------------------- 1 | 2 | 3 | import math 4 | import time 5 | import colorsys 6 | from . import libaudioverse 7 | import wx 8 | from .screenBitmap import rgbPixelBrightness 9 | 10 | fadeLength=0.05 11 | sweepGap=0.2 12 | maxBrightness=255 13 | 14 | class ImagePlayer_pitchStereoGrey(object): 15 | 16 | reverseBrightness=False 17 | sweepDuration=4 18 | _sweeperCallback=None 19 | 20 | def __init__(self,width,height,lowFreq=500,highFreq=5000,sweepDelay=0.5,sweepDuration=4,sweepCount=4,reverseBrightness=False): 21 | self.width=width 22 | self.height=height 23 | self.baseFreq=lowFreq 24 | self.octiveCount=math.log(highFreq/lowFreq,2) 25 | self.sweepDelay=sweepDelay 26 | self.sweepDuration=sweepDuration 27 | self.sweepCount=sweepCount 28 | self.reverseBrightness=reverseBrightness 29 | self.lavServer=libaudioverse.Server() 30 | self.lavPanner=libaudioverse.MultipannerNode(self.lavServer,"default") 31 | self.lavPanner.strategy=libaudioverse.PanningStrategies.hrtf 32 | self.lavPanner.should_crossfade=False 33 | self.lavPanner.mul=0 34 | self.lavPanner.connect(0,self.lavServer) 35 | self.lavWaves=[] 36 | for x in range(self.height): 37 | lavPanner=libaudioverse.AmplitudePannerNode(self.lavServer) 38 | lavPanner.mul=0 39 | lavPanner.should_crossfade=False 40 | lavPanner.connect(0,self.lavServer) 41 | lavWave=libaudioverse.SineNode(self.lavServer) 42 | lavWave.mul=0 43 | lavWave.frequency.value=self.baseFreq*((2**self.octiveCount)**(x/self.height)) 44 | lavWave.connect(0,lavPanner,0) 45 | lavWave.connect(0,self.lavPanner,0) 46 | self.lavWaves.append((lavWave,lavPanner)) 47 | self.lavServer.set_output_device("default") 48 | 49 | def _playWholeImage(self,imageData): 50 | self.lavPanner.azimuth.value=self.lavPanner.azimuth.value 51 | self.lavPanner.azimuth.linear_ramp_to_value(fadeLength,0) 52 | self.lavPanner.mul.value=self.lavPanner.mul.value 53 | self.lavPanner.mul.linear_ramp_to_value(fadeLength,0) 54 | totalVolume=0 55 | for y in range(self.height): 56 | index=-1-y; 57 | lavWave,lavPanner=self.lavWaves[index] 58 | left=0 59 | right=0 60 | brightest=0 61 | for x in range(self.width): 62 | rRatio=x/self.width 63 | lRatio=1-rRatio 64 | px=rgbPixelBrightness(imageData[y][x]) 65 | if self.reverseBrightness: 66 | px=maxBrightness-px 67 | brightest=max(brightest,px) 68 | left+=px*lRatio 69 | right+=px*rRatio 70 | volume=brightest/maxBrightness 71 | lavWave.mul.value=lavWave.mul.value 72 | lavWave.mul.linear_ramp_to_value(fadeLength,volume) 73 | totalVolume+=volume 74 | waveAngle=((right-left)/max(left,right))*90 if (left or right) else 0 75 | lavPanner.azimuth.value=lavPanner.azimuth.value 76 | lavPanner.azimuth.linear_ramp_to_value(fadeLength,waveAngle) 77 | volumeRatio=0.075 if totalVolume<=1.0 else 0.075/totalVolume 78 | for y in range(self.height): 79 | lavWave,lavPanner=self.lavWaves[y] 80 | lavPanner.mul.value=lavPanner.mul.value 81 | lavPanner.mul.linear_ramp_to_value(fadeLength,volumeRatio) 82 | 83 | def _sweepImage(self,imageData,duration,count): 84 | offset=0 85 | totalVolumes=[0]*self.width 86 | for y in range(self.height): 87 | index=-1-y; 88 | lavWave,lavPanner=self.lavWaves[index] 89 | lavPanner.mul=0 90 | lavWave.mul=0 91 | envelopeValues=[0] 92 | for x in range(self.width): 93 | px=rgbPixelBrightness(imageData[y][x]) 94 | if self.reverseBrightness: 95 | px=maxBrightness-px 96 | volume=px/maxBrightness 97 | envelopeValues.append(volume) 98 | envelopeValues.append(0) 99 | totalVolumes[x]+=volume 100 | offset=0 101 | for c in range(count): 102 | lavWave.mul.set(offset,0) 103 | offset+=sweepGap 104 | lavWave.mul.envelope(time=offset,duration=duration,values=envelopeValues) 105 | offset+=duration 106 | for index,totalVolume in enumerate(totalVolumes): 107 | totalVolumes[index]=0.075 if totalVolume<=1.0 else 0.075/totalVolume 108 | self.lavPanner.azimuth=-90 109 | self.lavPanner.mul=0 110 | offset=0 111 | for c in range(count): 112 | self.lavPanner.azimuth.set(offset,-90) 113 | self.lavPanner.mul.set(offset,0) 114 | offset+=sweepGap 115 | self.lavPanner.azimuth.envelope(time=offset,duration=duration,values=list(range(-90,91))) 116 | self.lavPanner.mul.envelope(time=offset,duration=duration,values=totalVolumes) 117 | offset+=duration 118 | 119 | def _stop(self): 120 | self.lavPanner.azimuth.value=0 121 | for y in range(self.height): 122 | lavWave=self.lavWaves[y][0] 123 | lavWave.mul.value=lavWave.mul.value 124 | lavWave.mul.linear_ramp_to_value(fadeLength,0) 125 | 126 | def setNewImage(self,imageData,detailed=False): 127 | if self._sweeperCallback: 128 | self._sweeperCallback.Stop() 129 | with self.lavServer: 130 | if not imageData: 131 | self._stop() 132 | else: 133 | if not detailed: 134 | self._playWholeImage(imageData) 135 | self._sweeperCallback=wx.CallLater(self.sweepDelay*1000,self._sweepImage,imageData,self.sweepDuration,self.sweepCount) 136 | else: 137 | self._sweepImage(imageData,self.sweepDuration,self.sweepCount) 138 | 139 | def terminate(self): 140 | self.setNewImage(None) 141 | self.lavServer.clear_output_device() 142 | 143 | class ImagePlayer_hsv(object): 144 | 145 | def __init__(self,width,height,lowFreq=90,highFreq=4000): 146 | self.width=width 147 | self.height=height 148 | self.lowFreq=lowFreq 149 | self.highFreq=highFreq 150 | self.lavServer=libaudioverse.Server() 151 | self.lavWave=libaudioverse.AdditiveSawNode(self.lavServer) 152 | self.lavWave.mul=0 153 | self.lavWave.frequency.value=lowFreq 154 | self.lavWave.connect(0,self.lavServer) 155 | self.lavWave2=libaudioverse.SineNode(self.lavServer) 156 | self.lavWave2.mul=0 157 | self.lavWave2.frequency.value=lowFreq*(highFreq/lowFreq) 158 | self.lavWave2.connect(0,self.lavServer) 159 | self.lavNoise=libaudioverse.NoiseNode(self.lavServer) 160 | self.lavNoise.mul.value=0 161 | self.lavNoise.noise_type.value=libaudioverse.NoiseTypes.brown 162 | self.lavNoise.connect(0,self.lavServer) 163 | self.lavServer.set_output_device("default") 164 | 165 | def setNewImage(self,imageData,detailed=False): 166 | r=g=b=0 167 | if imageData is not None: 168 | for x in range(self.height): 169 | for y in range(self.width): 170 | px=imageData[y][x] 171 | r+=px.rgbRed 172 | g+=px.rgbGreen 173 | b+=px.rgbBlue 174 | r/=(self.width*self.height) 175 | g/=(self.width*self.height) 176 | b/=(self.width*self.height) 177 | h,s,v=colorsys.rgb_to_hsv(r/255,g/255,b/255) 178 | s=1-(10**(1-s)/10) 179 | iH=1-h 180 | iH_fromBlue=min(max(iH-0.333,0)/0.666,1) 181 | iH_imag=min(iH/0.333,1) 182 | self.lavWave.mul.value=v*s*iH_imag*0.75/(1+(iH_fromBlue*10)) 183 | self.lavWave.frequency.value=self.lowFreq*((self.highFreq/self.lowFreq)**((2**iH_fromBlue)-1)) 184 | self.lavWave.harmonics=int(1+((((1-abs(iH_fromBlue-0.5))*2)-1)*20)) 185 | self.lavWave2.mul.value=v*s*(1-iH_imag)*0.075 186 | self.lavNoise.mul.value=(1-s)*v*0.4 187 | 188 | def terminate(self): 189 | self.lavServer.clear_output_device() 190 | -------------------------------------------------------------------------------- /addon/globalPlugins/audioScreen/screenBitmap.py: -------------------------------------------------------------------------------- 1 | import ctypes 2 | import winGDI 3 | 4 | user32=ctypes.windll.user32 5 | gdi32=ctypes.windll.gdi32 6 | 7 | class ScreenBitmap(object): 8 | """Provides a way to capture a bitmap of any part of the screen. The object caches needed DCs and bitmaps therefore an instance of an object only handles one size of bitmap.""" 9 | 10 | def __init__(self,width,height): 11 | """ 12 | @param width: the width of the resulting bitmap in rgb pixels. 13 | @param height: the height of the bitmap in rgb pixels. 14 | """ 15 | self.width=width 16 | self.height=height 17 | #Fetch the device context for the screen 18 | self._screenDC=user32.GetDC(0) 19 | #Create a memory device context with which we can copy screen content to on request. 20 | self._memDC=gdi32.CreateCompatibleDC(self._screenDC) 21 | #Create a new bitmap of the chosen size, and set this as the memory device context's bitmap, so that what is drawn is captured. 22 | self._memBitmap=gdi32.CreateCompatibleBitmap(self._screenDC,width,height) 23 | self._oldBitmap=gdi32.SelectObject(self._memDC,self._memBitmap) 24 | #We always want standard RGB data 25 | bmInfo=winGDI.BITMAPINFO() 26 | bmInfo.bmiHeader.biSize=ctypes.sizeof(bmInfo) 27 | bmInfo.bmiHeader.biWidth=width 28 | bmInfo.bmiHeader.biHeight=height*-1 29 | bmInfo.bmiHeader.biPlanes=1 30 | bmInfo.bmiHeader.biBitCount=32 31 | bmInfo.bmiHeader.biCompression=winGDI.BI_RGB 32 | self._bmInfo=bmInfo 33 | gdi32.SetStretchBltMode(self._memDC,4) 34 | 35 | def __del__(self): 36 | gdi32.SelectObject(self._memDC,self._oldBitmap) 37 | gdi32.DeleteObject(self._memBitmap) 38 | gdi32.DeleteDC(self._memDC) 39 | user32.ReleaseDC(0,self._screenDC) 40 | 41 | def captureImage(self,x,y,w,h): 42 | """ 43 | Captures the part of the screen starting at x,y and extends by w (width) and h (height), and stretches/shrinks it to fit in to the object's bitmap size. 44 | """ 45 | #Copy the requested content from the screen in to our memory device context, stretching/shrinking its size to fit. 46 | gdi32.StretchBlt(self._memDC,0,0,self.width,self.height,self._screenDC,int(x),int(y),int(w),int(h),winGDI.SRCCOPY) 47 | #Fetch the pixels from our memory bitmap and store them in a buffer to be returned 48 | buffer=(winGDI.RGBQUAD*self.width*self.height)() 49 | gdi32.GetDIBits(self._memDC,self._memBitmap,0,self.height,buffer,ctypes.byref(self._bmInfo),winGDI.DIB_RGB_COLORS); 50 | return buffer 51 | 52 | def rgbPixelBrightness(p): 53 | """Converts a RGBQUAD pixel in to one grey-scale brightness value.""" 54 | return int((0.3*p.rgbBlue)+(0.59*p.rgbGreen)+(0.11*p.rgbRed)) 55 | -------------------------------------------------------------------------------- /addon/manifest.ini: -------------------------------------------------------------------------------- 1 | name=audioScreen 2 | summary=Audio Screen 3 | version=2020.1 4 | author=NV Access Limited 5 | description="""Allows you to "feel" images with your ears, while moving your finger around a touch screen on Windows 8 and above.""" 6 | url=http://www.github.com/nvaccess/audioScreen 7 | minimumNVDAVersion = 2019.3.0 8 | lastTestedNVDAVersion = 2019.3.1 9 | updateChannel = None 10 | -------------------------------------------------------------------------------- /readme.md: -------------------------------------------------------------------------------- 1 | # AudioScreen: an experiment inimage accessibility for blind people 2 | By Michael Curran, NV Access Limited 3 | ## Introduction 4 | Audio Screen is an add-on for the [NVDA Screen Reading software](http://www.nvaccess.org/). Audio Screen can allow a blind person to move their finger around a Windows 8+ compatible touch screen, and hear the part of the image under their finger. If a touch screen is not available, the mouse can be moved instead, though this is some what less accurate for the user as mouse movement is relative. 5 | 6 | Here a [demonstration of AudioScreen by Michael Curran [mp3 file, 17 mb]](http://www.nvaccess.org/audioScreen/audioScreenDemo20151129.mp3) where he demonstrates the various modes, and uses it to explore a map of Australia, a rainbow, the earth from space, a cartoon house, and a sun set. 7 | 8 | As Audio Screen requires NVDA to run, the user will therefore also receive speech feedback such as the name of the control or text, directly under their finger. 9 | 10 | Audio Screen can be seen as an experimental alternative way for blind people to view basic images such as diagrams and maps when no other tactile format is available. 11 | 12 | AudioScreen has two modes of output: "pitch stereo grey" for investigating lines and contours of images (useful for maps and diagrams etc), and "HSV color" for investigating the color variation of images (useful for photographs). 13 | 14 | ## Pitch Stereo Grey mode 15 | 16 | In this mode, AudioScreen represents the image under your finger or the mouse as multiple tones that vary in pitch, volume and stereo position. The idea is based on the vOICe visual-to-auditory mapping system by Peter Meijor. (www.seeingwithsound.com) 17 | audio Screen strives to provide roughly the same information your finger would for tactile diagrams. You can tell when your finger or mouse moves over a line, both horizontally and vertically. 18 | For example if your finger crosses a horizontal line as it moves down the screen, you can hear the line move up over your finger. If your finger crosses a vertical line as you move across the screen from left to right, you will hear the line move across your finger from right to left. 19 | If you leave your finger or mouse stationary at a point on the screen for more than half a second, audioScreen will start sweeping the audio from left to right, isolating single columns of pixels, providing much more extreme detail of the image to help with detecting patterns etc. 20 | 21 | In NVDA 2016.1 or later, If you place more than one finger on the screen at a time, audioScreen will sweep over the image bounded by all your fingers. For example, placing one finger at the top left of a large image, and one fingr at the bottom right, audioScreen will sweep over the entire image. 22 | 23 | A part from receiving feedback from touch input or a mouse, you can also instruct audioScreen to sweep over an entire image or control, via its play navigator object command. This will perform multiple vOICe-style sweeps over NVDA's current navigator object. 24 | 25 | ## HSV Color mode 26 | 27 | In this mode, AudioScreen will convey the color (specifically hue, saturation and brightness) of the image under your finger. 28 | 29 | Hue (position in the color spectrum) is represented by a tone at a particular pitch. Currently, the lowest pitch is blue, moving up through green, yellow, orange, red (the highest pitch). As a spectrum is infact a circle, going from red, through purples back to blue, the top (red) pitch fades out, and the bottom (blue) pitch fades back in. 30 | 31 | Saturation (how vivid the color is) is represented by brown noise (low random noise). When the color is its most vivid (full saturation) there is no random noise and the spectrum tone can be completely heard. As the saturation decreases towards grey (no saturation) the spectrum tone gets quieter and the random noise gets louder, eventually with only the random noise being heard at no saturation. 32 | 33 | Brightness is represented by the over all volume of the sound as a whole. 34 | 35 | Some examples: 36 | * black: silence 37 | * White: loud random noise 38 | * Vivid blue: a very low tone 39 | * Vivid yellow: a mid to high tone 40 | * Faded red: a very high tone with some random noise. 41 | * Dark green: a quiet low-mid tone. 42 | * Dark grey: quiet random noise. 43 | 44 | ## System requirements 45 | * An installed copy of NVDA 2015.4 or higher (NVDA 2016.1 or higher for multi-finger sweeping) 46 | * Windows 8 Operating system or later 47 | * A Windows 8/10 compatible touch screen, otherwise a mouse. 48 | * Visual feedback for touch must be turned off in Windows. Search for Change Touch Input setting in the start screen, and in that dialog uncheck Show visual feedback when touching the screen. 49 | 50 | ## Download 51 | * Download [AudioScreen 2020.1 [NVDA add-on file]](http://www.nvaccess.org/audioScreen/audioScreen-2020.1.nvda-addon). 52 | * Download [Example images [zip file]](http://www.nvaccess.org/audioScreen/audioScreenImages.zip). 53 | 54 | ## Running AudioScreen 55 | After downloading the Audio Screen add-on, with NVDA running simply press enter on the file in Windows Explorer to have NVDA install it for you. Alternatively, you can open Manage Add-ons found under Tools in the NVDA menu, and add it from there. After the add-on is installed, NVDA will ask to be restarted. 56 | 57 | Important: Visual feedback for touch must be turned off in Windows. Search for Change Touch Input setting in the start screen, and in that dialog uncheck Show visual feedback when touching the screen. 58 | 59 | While NVDA is running with this add-on installed, open an interesting image in full-screen (For example, use Internet Explorer to display one of the example svg files, making sure to maximize it and put it in full-screen with f11). 60 | 61 | AudioScreen is off by default, so turn it on by switching to one of its modes by pressing NVDA+control+a. This toggles between Pitch stereo grey, HSV color, and off. 62 | 63 | Now move your finger or mouse around the screen and start listening to the image under your finger. 64 | As NVDA can also speak controls and text under your finger, viewing an SVG map or diagram works great, as Internet Explorer will allow NVDA to speak the title and or description for any shape your finger moves over, assuming that a title and or description have been properly defined using the title and desc tags appropriately in the SVG file. 65 | 66 | ## Commands 67 | ### Change Audio Mode (Press NVDA+Control+a) 68 | This command toggles between several modes: 69 | * pitch stereo grey: for investigating lines and contours of images (useful for maps and diagrams etc) 70 | * HSV color: for investigating the color variation of images (useful for photographs). 71 | * Off [default]: completely disables AudioScreen. 72 | 73 | ### Play Navigator Object (Press NVDA+alt+a) 74 | This command will play NVDA's current navigator object, by performing multiple vOICe-style stereo sweeps across it. 75 | 76 | ### Show Settings UI (Press NVDA+shift+a) 77 | This brings up a settings dialog which allows you to change multiple options for audioScreen. The Settings UI can also be launched by choosing AudioScreen... found under Preferences in the NVDA menu. 78 | 79 | ## Settings 80 | 81 | ### AudioScreen Mode 82 | Choose the desired mode: 83 | * pitch stereo grey: for investigating lines and contours of images (useful for maps and diagrams etc) 84 | * HSV color: for investigating the color variation of images (useful for photographs). 85 | * Off [default]: completely disables AudioScreen. 86 | 87 | ### Pitch Stereo Grey settings 88 | 89 | #### Reverse brightness 90 | This option allows you to reverse the brightness of the image, so that rather than light being loud and dark being quiet, dark will be loud and light will be quiet. Very useful when playing an image where the foreground objects are darker than the background. 91 | 92 | #### Number of columns in stereo field 93 | How wide (in pixels) the image should be. When moving with your finger or the mouse, this is literally how wide the captured image is. For play navigator object, although the full image is fetched, it is compressed or stretched to fit this width. 94 | 95 | #### Number of rows (frequencies) 96 | How tall (in pixels) the image should be. Each row of pixels is represented by a particular frequency. frequencies are spread out logarithmically. When moving with your finger or the mouse, this is literally how tall the captured image is. For play navigator object, although the full image is fetched, it is compressed or stretched to fit this height. 97 | 98 | #### Lowest frequency in HZ 99 | The frequency (in HZ) used for the bottom most row of the image. 100 | 101 | #### Highest frequency in HZ 102 | The frequency (in HZ) Used for the top most row of the image. 103 | 104 | #### Initial stereo sweep delay in seconds 105 | How long audioScreen should wait (in seconds) to start sweeping an image, after your finger or the mouse has moved. 106 | 107 | #### Duration of stereo audio sweep in seconds 108 | How long (in seconds) each sweep should go for. 109 | 110 | #### Number of stereo sweeps 111 | The number of sweeps that should be played once your finger or the mouse has moved, or when the play navigator object command is run. 112 | 113 | #### Width (in pixels) of the rectangle at the point under your finger / the mouse 114 | How wide is the capture area when playing a single point on the screen. 115 | 116 | #### Height (in pixels) of the rectangle at the point under your finger / the mouse 117 | How tall is the capture area when playing a single point on the screen. 118 | 119 | ### HSV Color settings 120 | 121 | #### Horizontal length of capture area in pixels 122 | The width (in pixels) of the area under your finger or the mouse captured to detect the color. The color is averaged over this area. Smaller values will give more accurate colors, though can cause you to hear more detail than perhaps seen visually. 123 | 124 | #### Vertical length of capture area in pixels 125 | The height (in pixels) of the area under your finger or the mouse captured to detect the color. The color is averaged over this area. Smaller values will give more accurate colors, though can cause you to hear more detail than perhaps seen visually. 126 | 127 | #### Lowest frequency (blue) in HZ 128 | The frequency (in HZ) that represents blue. The frequency rises through aqua, green, yellow, orange, to red. As the color spectrum raps around from red back to blue through purple, purples are represented by both the low (blue) frequency and high (red) frequency at differing volume ratios. I.e. A blue-ish purple will be mostly the low (blue) frequency with a small amount of the high (red) frequency). 129 | 130 | 131 | #### highest frequency (red) in HZ 132 | The frequency (in HZ) that represents red. The frequency falls through orange, yellow, green, aqua, to blue. As the color spectrum raps around from blue back to red through purple, purples are represented by both the low (blue) frequency and high (red) frequency at differing volume ratios. I.e. A red-ish purple will be mostly the high (red) frequency with a small amount of the low (blue) frequency. 133 | 134 | ## Developing and Packaging from source 135 | 136 | Clone the AudioScreen repository with the command: 137 | git clone https://www.github.com/nvaccess/audioScreen 138 | 139 | [Python 2.7](http://www.python.org/) is required for building and developing this project. 140 | 141 | AudioScreen depends on [libaudioverse 0.8](https://www.github.com/camlorn/libaudioverse): 142 | * cd to the audioScreen repository you cloned with git 143 | * Run the command: pip install --ignore-installed -t addon\globalPlugins\audioScreen\deps libaudioverse 144 | 145 | ### Packaging the NVDA add-on 146 | In the addon directory: 147 | * Edit manifest.ini to set the version of the add-on etc. 148 | * Create a zip file using your favorite zip tool, including both manifest.ini and the globalPlugins directory. It is important that both manifest.ini and globalPlugins be in the root of the zip file, I.e. no interviening directory. The zip file should have a.nvda-addon extension. 149 | 150 | ## Background 151 | For quite some time now, I have wanted a way to get access as a blind person to maps and basic diagrams with out the hassles of having to produce them in a tactile format. 152 | 153 | Although tactile formats are certainly very useful when they are available, they do have particular drawbacks, such as: 154 | * They are slow to produce 155 | * Special materials are needed 156 | * Sometimes a special machine is needed 157 | * Rather non-environmental due to paper wastage 158 | * Bulky to carry around 159 | * Only very limited labeling can be included due to space constraints 160 | 161 | A few years ago, I stumbled upon a very interesting peace of software called The vOICe, from www.seeingwithsound.com. This software could take images or video from a webcam, and play it to a blind person as audio, making use of pitch, volume and stereo panning. Although the vOICe software is more passive in the sence that it scans from left to right across an entire image, I could see many possibilities of using this image-to-sound mapping concept in a more active way, with the help of a touch screen. 162 | 163 | I should also note that I am aware of other research in to conveying images on touch screens with sound, but none of them that I have seen so far, have yet chosen to try using the vOICe mapping concept. 164 | 165 | When conveying information from one sence modality to another, I believe its very important not to loose information in the process. If you can provide roughly the same or better resolution in the second sence, the brain will have a much easier time of decoding the information. A mapping such as the vOICe I believe certainly gets extremely close to achieving this. 166 | 167 | ### Mapping color to sound 168 | Although access to basic diagrams such as maps and other line-based drawings have many practical applications for the blind, there is also an argument that access to color images such as in art or the beauty of the world, has some subjective importance. For example how colors vary in a rainbow, or a picture of the earth from space. These things are very hard to describe in words. 169 | --------------------------------------------------------------------------------