├── requirements-dev.txt ├── docs ├── _build │ ├── html │ │ ├── objects.inv │ │ ├── _static │ │ │ ├── up.png │ │ │ ├── down.png │ │ │ ├── file.png │ │ │ ├── minus.png │ │ │ ├── plus.png │ │ │ ├── comment.png │ │ │ ├── up-pressed.png │ │ │ ├── ajax-loader.gif │ │ │ ├── down-pressed.png │ │ │ ├── comment-bright.png │ │ │ ├── comment-close.png │ │ │ ├── pygments.css │ │ │ ├── default.css │ │ │ ├── sidebar.js │ │ │ ├── doctools.js │ │ │ ├── basic.css │ │ │ └── underscore.js │ │ ├── _sources │ │ │ ├── JSGFParser.txt │ │ │ ├── JSGFGrammar.txt │ │ │ ├── DeterministicGenerator.txt │ │ │ ├── ProbabilisticGenerator.txt │ │ │ └── index.txt │ │ ├── .buildinfo │ │ ├── search.html │ │ ├── searchindex.js │ │ ├── py-modindex.html │ │ ├── index.html │ │ ├── JSGFGrammar.html │ │ ├── DeterministicGenerator.html │ │ ├── ProbabilisticGenerator.html │ │ ├── genindex.html │ │ └── JSGFParser.html │ └── doctrees │ │ ├── index.doctree │ │ ├── JSGFParser.doctree │ │ ├── environment.pickle │ │ ├── JSGFGrammar.doctree │ │ ├── DeterministicGenerator.doctree │ │ └── ProbabilisticGenerator.doctree ├── JSGFParser.rst ├── JSGFGrammar.rst ├── DeterministicGenerator.rst ├── ProbabilisticGenerator.rst ├── index.rst ├── Makefile └── conf.py ├── Ideas.gram ├── pytest.ini ├── MANIFEST.in ├── IdeasNonRecursive.gram ├── .gitignore ├── jsgf ├── __init__.py ├── exceptions.py ├── legacy_adapter.py ├── ast_nodes.py ├── cli.py ├── grammar.py └── parser.py ├── LICENSE ├── pyproject.toml ├── setup.py ├── JSGFGrammar.py ├── README.md ├── DeterministicGenerator.py ├── ProbabilisticGenerator.py └── JSGFParser.py /requirements-dev.txt: -------------------------------------------------------------------------------- 1 | # Development dependencies for JSGFTools 2 | pytest>=7.0.0 3 | pyparsing>=3.0.0 -------------------------------------------------------------------------------- /docs/_build/html/objects.inv: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/syntactic/JSGFTools/HEAD/docs/_build/html/objects.inv -------------------------------------------------------------------------------- /docs/_build/html/_static/up.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/syntactic/JSGFTools/HEAD/docs/_build/html/_static/up.png -------------------------------------------------------------------------------- /docs/_build/doctrees/index.doctree: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/syntactic/JSGFTools/HEAD/docs/_build/doctrees/index.doctree -------------------------------------------------------------------------------- /docs/_build/html/_static/down.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/syntactic/JSGFTools/HEAD/docs/_build/html/_static/down.png -------------------------------------------------------------------------------- /docs/_build/html/_static/file.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/syntactic/JSGFTools/HEAD/docs/_build/html/_static/file.png -------------------------------------------------------------------------------- /docs/_build/html/_static/minus.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/syntactic/JSGFTools/HEAD/docs/_build/html/_static/minus.png -------------------------------------------------------------------------------- /docs/_build/html/_static/plus.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/syntactic/JSGFTools/HEAD/docs/_build/html/_static/plus.png -------------------------------------------------------------------------------- /docs/_build/html/_static/comment.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/syntactic/JSGFTools/HEAD/docs/_build/html/_static/comment.png -------------------------------------------------------------------------------- /docs/_build/doctrees/JSGFParser.doctree: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/syntactic/JSGFTools/HEAD/docs/_build/doctrees/JSGFParser.doctree -------------------------------------------------------------------------------- /docs/_build/doctrees/environment.pickle: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/syntactic/JSGFTools/HEAD/docs/_build/doctrees/environment.pickle -------------------------------------------------------------------------------- /docs/_build/html/_static/up-pressed.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/syntactic/JSGFTools/HEAD/docs/_build/html/_static/up-pressed.png -------------------------------------------------------------------------------- /docs/_build/doctrees/JSGFGrammar.doctree: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/syntactic/JSGFTools/HEAD/docs/_build/doctrees/JSGFGrammar.doctree -------------------------------------------------------------------------------- /docs/_build/html/_static/ajax-loader.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/syntactic/JSGFTools/HEAD/docs/_build/html/_static/ajax-loader.gif -------------------------------------------------------------------------------- /docs/_build/html/_static/down-pressed.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/syntactic/JSGFTools/HEAD/docs/_build/html/_static/down-pressed.png -------------------------------------------------------------------------------- /Ideas.gram: -------------------------------------------------------------------------------- 1 | public = ; 2 | 3 | = /5/ the idea | /1/ the idea ; 4 | = will suffice; 5 | 6 | = that ; 7 | -------------------------------------------------------------------------------- /docs/JSGFParser.rst: -------------------------------------------------------------------------------- 1 | JSGFParser module 2 | ================== 3 | 4 | .. automodule:: JSGFParser 5 | :members: 6 | :undoc-members: 7 | -------------------------------------------------------------------------------- /docs/_build/html/_static/comment-bright.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/syntactic/JSGFTools/HEAD/docs/_build/html/_static/comment-bright.png -------------------------------------------------------------------------------- /docs/_build/html/_static/comment-close.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/syntactic/JSGFTools/HEAD/docs/_build/html/_static/comment-close.png -------------------------------------------------------------------------------- /docs/JSGFGrammar.rst: -------------------------------------------------------------------------------- 1 | JSGFGrammar module 2 | ================== 3 | 4 | .. automodule:: JSGFGrammar 5 | :members: 6 | :undoc-members: 7 | -------------------------------------------------------------------------------- /docs/_build/doctrees/DeterministicGenerator.doctree: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/syntactic/JSGFTools/HEAD/docs/_build/doctrees/DeterministicGenerator.doctree -------------------------------------------------------------------------------- /docs/_build/doctrees/ProbabilisticGenerator.doctree: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/syntactic/JSGFTools/HEAD/docs/_build/doctrees/ProbabilisticGenerator.doctree -------------------------------------------------------------------------------- /pytest.ini: -------------------------------------------------------------------------------- 1 | [tool:pytest] 2 | testpaths = . 3 | python_files = test_*.py 4 | python_classes = Test* 5 | python_functions = test_* 6 | addopts = -v --tb=short -------------------------------------------------------------------------------- /docs/_build/html/_sources/JSGFParser.txt: -------------------------------------------------------------------------------- 1 | JSGFParser module 2 | ================== 3 | 4 | .. automodule:: JSGFParser 5 | :members: 6 | :undoc-members: 7 | -------------------------------------------------------------------------------- /docs/_build/html/_sources/JSGFGrammar.txt: -------------------------------------------------------------------------------- 1 | JSGFGrammar module 2 | ================== 3 | 4 | .. automodule:: JSGFGrammar 5 | :members: 6 | :undoc-members: 7 | -------------------------------------------------------------------------------- /docs/DeterministicGenerator.rst: -------------------------------------------------------------------------------- 1 | Deterministic Generator module 2 | ============================== 3 | 4 | .. automodule:: DeterministicGenerator 5 | :members: 6 | :undoc-members: 7 | -------------------------------------------------------------------------------- /docs/ProbabilisticGenerator.rst: -------------------------------------------------------------------------------- 1 | Probabilistic Generator module 2 | ============================== 3 | 4 | 5 | .. automodule:: ProbabilisticGenerator 6 | :members: 7 | :undoc-members: 8 | -------------------------------------------------------------------------------- /docs/_build/html/_sources/DeterministicGenerator.txt: -------------------------------------------------------------------------------- 1 | Deterministic Generator module 2 | ============================== 3 | 4 | .. automodule:: DeterministicGenerator 5 | :members: 6 | :undoc-members: 7 | -------------------------------------------------------------------------------- /docs/_build/html/_sources/ProbabilisticGenerator.txt: -------------------------------------------------------------------------------- 1 | Probabilistic Generator module 2 | ============================== 3 | 4 | 5 | .. automodule:: ProbabilisticGenerator 6 | :members: 7 | :undoc-members: 8 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | include README.md 2 | include LICENSE 3 | include CLAUDE.md 4 | include requirements-dev.txt 5 | include pytest.ini 6 | include *.gram 7 | recursive-include jsgf *.py 8 | recursive-exclude * __pycache__ 9 | recursive-exclude * *.py[co] 10 | -------------------------------------------------------------------------------- /docs/_build/html/.buildinfo: -------------------------------------------------------------------------------- 1 | # Sphinx build info version 1 2 | # This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done. 3 | config: 788090ed81524437c57a51ef201002b0 4 | tags: 645f666f9bcd5a90fca523b33c5a78b7 5 | -------------------------------------------------------------------------------- /IdeasNonRecursive.gram: -------------------------------------------------------------------------------- 1 | public = | ; 2 | 3 | = /5/ the idea | /1/ the idea ; 4 | = suffice; 5 | 6 | = that ; 7 | 8 | = ; 9 | 10 | = the idea; 11 | 12 | = /10/ can | /15/ should | /5/ might; 13 | 14 | = the idea can suffice; 15 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Python 2 | *.pyc 3 | __pycache__/ 4 | *.pyo 5 | *.pyd 6 | .Python 7 | build/ 8 | develop-eggs/ 9 | dist/ 10 | downloads/ 11 | eggs/ 12 | .eggs/ 13 | lib/ 14 | lib64/ 15 | parts/ 16 | sdist/ 17 | var/ 18 | wheels/ 19 | *.egg-info/ 20 | .installed.cfg 21 | *.egg 22 | 23 | # Testing 24 | .pytest_cache/ 25 | .coverage 26 | htmlcov/ 27 | 28 | # IDEs 29 | .vscode/ 30 | .idea/ 31 | *.swp 32 | *.swo 33 | 34 | # OS 35 | .DS_Store 36 | Thumbs.db 37 | 38 | # Virtual environments 39 | venv/ 40 | env/ 41 | ENV/ 42 | -------------------------------------------------------------------------------- /jsgf/__init__.py: -------------------------------------------------------------------------------- 1 | """ 2 | JSGF Tools - Modern Python library for parsing and generating from JSGF grammars. 3 | 4 | This package provides a clean, object-oriented API for working with JSGF grammars. 5 | """ 6 | 7 | from .grammar import Grammar 8 | from .generators import DeterministicGenerator, ProbabilisticGenerator 9 | from .exceptions import JSGFError, ParseError, GenerationError 10 | 11 | __version__ = "2.0.0" 12 | __all__ = [ 13 | "Grammar", 14 | "DeterministicGenerator", 15 | "ProbabilisticGenerator", 16 | "JSGFError", 17 | "ParseError", 18 | "GenerationError" 19 | ] -------------------------------------------------------------------------------- /docs/index.rst: -------------------------------------------------------------------------------- 1 | .. JSGF Grammar Tools documentation master file, created by 2 | sphinx-quickstart on Thu May 29 15:47:59 2014. 3 | You can adapt this file completely to your liking, but it should at least 4 | contain the root `toctree` directive. 5 | 6 | Welcome to JSGF Grammar Tools's documentation! 7 | ============================================== 8 | 9 | This set of tools is intended to aid the manipulation of 10 | JSGF Grammars. It includes a parser, and two string generators which 11 | can be used to create artificial corpora. Some additional tools that will 12 | be added in the near future include a tool to check grammars for recursion 13 | and a tool to calculate the total number of strings a grammar generates. 14 | 15 | Contents: 16 | 17 | .. toctree:: 18 | :maxdepth: 2 19 | 20 | JSGFGrammar 21 | JSGFParser 22 | ProbabilisticGenerator 23 | DeterministicGenerator 24 | 25 | 26 | 27 | Indices and tables 28 | ================== 29 | 30 | * :ref:`genindex` 31 | * :ref:`modindex` 32 | * :ref:`search` 33 | 34 | -------------------------------------------------------------------------------- /docs/_build/html/_sources/index.txt: -------------------------------------------------------------------------------- 1 | .. JSGF Grammar Tools documentation master file, created by 2 | sphinx-quickstart on Thu May 29 15:47:59 2014. 3 | You can adapt this file completely to your liking, but it should at least 4 | contain the root `toctree` directive. 5 | 6 | Welcome to JSGF Grammar Tools's documentation! 7 | ============================================== 8 | 9 | This set of tools is intended to aid the manipulation of 10 | JSGF Grammars. It includes a parser, and two string generators which 11 | can be used to create artificial corpora. Some additional tools that will 12 | be added in the near future include a tool to check grammars for recursion 13 | and a tool to calculate the total number of strings a grammar generates. 14 | 15 | Contents: 16 | 17 | .. toctree:: 18 | :maxdepth: 2 19 | 20 | JSGFGrammar 21 | JSGFParser 22 | ProbabilisticGenerator 23 | DeterministicGenerator 24 | 25 | 26 | 27 | Indices and tables 28 | ================== 29 | 30 | * :ref:`genindex` 31 | * :ref:`modindex` 32 | * :ref:`search` 33 | 34 | -------------------------------------------------------------------------------- /jsgf/exceptions.py: -------------------------------------------------------------------------------- 1 | """ 2 | Custom exceptions for JSGF Tools. 3 | """ 4 | 5 | from typing import Optional 6 | 7 | 8 | class JSGFError(Exception): 9 | """Base exception for all JSGF-related errors.""" 10 | pass 11 | 12 | 13 | class ParseError(JSGFError): 14 | """Raised when grammar parsing fails.""" 15 | 16 | def __init__(self, message: str, line: Optional[int] = None, column: Optional[int] = None): 17 | self.line = line 18 | self.column = column 19 | 20 | if line is not None: 21 | message = f"Line {line}: {message}" 22 | if column is not None: 23 | message = f"{message} (column {column})" 24 | 25 | super().__init__(message) 26 | 27 | 28 | class GenerationError(JSGFError): 29 | """Raised when string generation fails.""" 30 | pass 31 | 32 | 33 | class ValidationError(JSGFError): 34 | """Raised when grammar validation fails.""" 35 | pass 36 | 37 | 38 | class RecursionError(GenerationError): 39 | """Raised when infinite recursion is detected during generation.""" 40 | pass -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2018 syntactic 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /pyproject.toml: -------------------------------------------------------------------------------- 1 | [build-system] 2 | requires = ["setuptools>=61.0", "wheel"] 3 | build-backend = "setuptools.build_meta" 4 | 5 | [project] 6 | name = "jsgf-tools" 7 | version = "2.1.1" 8 | description = "Complete JSGF toolkit: parse, generate, and test speech grammars with Unicode support" 9 | readme = "README.md" 10 | requires-python = ">=3.7" 11 | license = {text = "MIT"} 12 | authors = [ 13 | {name = "Pastèque Ho", email = "timothyakho@gmail.com"} 14 | ] 15 | keywords = ["jsgf", "grammar", "speech recognition", "nlp", "parsing", "generation", "unicode", "testing"] 16 | classifiers = [ 17 | "Development Status :: 4 - Beta", 18 | "Intended Audience :: Developers", 19 | "License :: OSI Approved :: MIT License", 20 | "Operating System :: OS Independent", 21 | "Programming Language :: Python :: 3", 22 | "Programming Language :: Python :: 3.7", 23 | "Programming Language :: Python :: 3.8", 24 | "Programming Language :: Python :: 3.9", 25 | "Programming Language :: Python :: 3.10", 26 | "Programming Language :: Python :: 3.11", 27 | "Programming Language :: Python :: 3.12", 28 | "Programming Language :: Python :: 3.13", 29 | "Topic :: Software Development :: Libraries :: Python Modules", 30 | "Topic :: Software Development :: Testing", 31 | "Topic :: Text Processing :: Linguistic", 32 | "Natural Language :: Chinese (Simplified)", 33 | "Natural Language :: Japanese", 34 | "Natural Language :: Korean", 35 | "Natural Language :: Arabic", 36 | "Natural Language :: Russian", 37 | "Natural Language :: Hebrew", 38 | "Natural Language :: Greek", 39 | "Natural Language :: Hindi", 40 | ] 41 | dependencies = [ 42 | "pyparsing>=3.0.0" 43 | ] 44 | 45 | [project.optional-dependencies] 46 | dev = [ 47 | "pytest>=7.0.0", 48 | "pytest-cov>=3.0.0", 49 | ] 50 | 51 | [project.urls] 52 | Homepage = "https://github.com/syntactic/JSGFTools" 53 | Documentation = "https://github.com/syntactic/JSGFTools#readme" 54 | Repository = "https://github.com/syntactic/JSGFTools" 55 | Issues = "https://github.com/syntactic/JSGFTools/issues" 56 | 57 | [project.scripts] 58 | jsgf-deterministic = "DeterministicGenerator:main" 59 | jsgf-probabilistic = "ProbabilisticGenerator:main" 60 | 61 | [tool.pytest.ini_options] 62 | testpaths = ["."] 63 | python_files = ["test_*.py"] 64 | python_classes = ["Test*"] 65 | python_functions = ["test_*"] 66 | addopts = "-v --strict-markers" 67 | 68 | [tool.setuptools] 69 | py-modules = ["JSGFParser", "JSGFGrammar", "DeterministicGenerator", "ProbabilisticGenerator"] 70 | 71 | [tool.setuptools.packages.find] 72 | exclude = ["tests*", "docs*", "examples*"] 73 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | from setuptools import setup, find_packages 3 | 4 | with open("README.md", "r", encoding="utf-8") as fh: 5 | long_description = fh.read() 6 | 7 | setup( 8 | name='jsgf-tools', 9 | version='2.1.1', 10 | author='Pastèque Ho', 11 | author_email='timothyakho@gmail.com', 12 | description='Complete JSGF toolkit: parse, generate, and test speech grammars with Unicode support', 13 | long_description=long_description, 14 | long_description_content_type="text/markdown", 15 | url='https://github.com/syntactic/JSGFTools', 16 | project_urls={ 17 | 'Bug Tracker': 'https://github.com/syntactic/JSGFTools/issues', 18 | 'Documentation': 'https://github.com/syntactic/JSGFTools#readme', 19 | 'Source Code': 'https://github.com/syntactic/JSGFTools', 20 | }, 21 | packages=find_packages(exclude=['tests*', 'docs*']), 22 | py_modules=['JSGFParser', 'JSGFGrammar', 'DeterministicGenerator', 'ProbabilisticGenerator'], 23 | classifiers=[ 24 | "Development Status :: 4 - Beta", 25 | "Intended Audience :: Developers", 26 | "License :: OSI Approved :: MIT License", 27 | "Operating System :: OS Independent", 28 | "Programming Language :: Python :: 3", 29 | "Programming Language :: Python :: 3.7", 30 | "Programming Language :: Python :: 3.8", 31 | "Programming Language :: Python :: 3.9", 32 | "Programming Language :: Python :: 3.10", 33 | "Programming Language :: Python :: 3.11", 34 | "Programming Language :: Python :: 3.12", 35 | "Programming Language :: Python :: 3.13", 36 | "Topic :: Software Development :: Libraries :: Python Modules", 37 | "Topic :: Software Development :: Testing", 38 | "Topic :: Text Processing :: Linguistic", 39 | "Natural Language :: Chinese (Simplified)", 40 | "Natural Language :: Japanese", 41 | "Natural Language :: Korean", 42 | "Natural Language :: Arabic", 43 | "Natural Language :: Russian", 44 | "Natural Language :: Hebrew", 45 | "Natural Language :: Greek", 46 | "Natural Language :: Hindi", 47 | ], 48 | keywords='jsgf grammar speech recognition nlp parsing generation unicode testing', 49 | python_requires=">=3.7", 50 | install_requires=[ 51 | "pyparsing>=3.0.0", 52 | ], 53 | extras_require={ 54 | 'dev': [ 55 | 'pytest>=7.0.0', 56 | 'pytest-cov>=3.0.0', 57 | ], 58 | }, 59 | entry_points={ 60 | 'console_scripts': [ 61 | 'jsgf-deterministic=DeterministicGenerator:main', 62 | 'jsgf-probabilistic=ProbabilisticGenerator:main', 63 | ], 64 | }, 65 | ) -------------------------------------------------------------------------------- /jsgf/legacy_adapter.py: -------------------------------------------------------------------------------- 1 | """ 2 | Adapter to use the existing JSGFParser with the new Grammar architecture. 3 | 4 | This provides a bridge between the old parser and the new modern API. 5 | """ 6 | 7 | from typing import TextIO 8 | import sys 9 | import os 10 | 11 | # Add the parent directory to the path to import the legacy modules 12 | sys.path.insert(0, os.path.dirname(os.path.dirname(__file__))) 13 | 14 | import JSGFParser as legacy_parser 15 | import JSGFGrammar as legacy_grammar 16 | 17 | from .ast_nodes import ( 18 | JSGFNode, Terminal, NonTerminal, Sequence, Alternative, 19 | Optional as OptionalNode, Group, Rule 20 | ) 21 | from .exceptions import ParseError 22 | 23 | 24 | class LegacyAdapter: 25 | """Adapter to convert legacy grammar objects to new AST format.""" 26 | 27 | def parse_to_grammar(self, stream: TextIO, grammar: 'Grammar') -> None: 28 | """ 29 | Parse using the legacy parser and convert to new Grammar format. 30 | 31 | Args: 32 | stream: Text stream containing JSGF grammar 33 | grammar: Grammar object to populate 34 | """ 35 | try: 36 | # Use the legacy parser 37 | legacy_gram = legacy_parser.getGrammarObject(stream) 38 | 39 | # Create a set of public rule names for easy lookup 40 | public_rule_names = {rule.lhs.name for rule in legacy_gram.publicRules} 41 | 42 | # Convert all rules, marking public ones appropriately 43 | for rule in legacy_gram.rules: 44 | is_public = rule.lhs.name in public_rule_names 45 | converted_rule = self._convert_rule(rule, is_public=is_public) 46 | grammar.add_rule(converted_rule) 47 | 48 | except Exception as e: 49 | raise ParseError(f"Failed to parse grammar: {e}") 50 | 51 | def _convert_rule(self, legacy_rule, is_public: bool = False) -> Rule: 52 | """Convert a legacy rule to new Rule format.""" 53 | rule_name = legacy_rule.lhs.name 54 | expansion = self._convert_expansion(legacy_rule.rhs) 55 | 56 | return Rule( 57 | name=rule_name, 58 | expansion=expansion, 59 | is_public=is_public 60 | ) 61 | 62 | def _convert_expansion(self, rhs) -> JSGFNode: 63 | """Convert legacy RHS to new AST format.""" 64 | if isinstance(rhs, str): 65 | return Terminal(rhs) 66 | elif isinstance(rhs, list): 67 | if len(rhs) == 1: 68 | return self._convert_expansion(rhs[0]) 69 | else: 70 | # Convert list to sequence 71 | elements = [self._convert_expansion(item) for item in rhs] 72 | return Sequence(elements) 73 | elif isinstance(rhs, legacy_grammar.Disjunction): 74 | # Convert disjunction 75 | choices = [] 76 | for disjunct in rhs.disjuncts: 77 | if isinstance(disjunct, tuple): 78 | # Weighted choice 79 | node, weight = disjunct 80 | choices.append((self._convert_expansion(node), weight)) 81 | else: 82 | # Unweighted choice 83 | choices.append((self._convert_expansion(disjunct), 1.0)) 84 | return Alternative(choices) 85 | elif isinstance(rhs, legacy_grammar.Optional): 86 | return OptionalNode(self._convert_expansion(rhs.option)) 87 | elif isinstance(rhs, legacy_grammar.NonTerminal): 88 | return NonTerminal(rhs.name) 89 | else: 90 | # Fallback for unknown types 91 | return Terminal(str(rhs)) -------------------------------------------------------------------------------- /docs/_build/html/search.html: -------------------------------------------------------------------------------- 1 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | Search — JSGF Grammar Tools 1.0 documentation 10 | 11 | 12 | 13 | 14 | 23 | 24 | 25 | 26 | 27 | 28 | 31 | 32 | 33 | 34 | 35 | 36 | 37 |
38 |

Navigation

39 | 48 |
49 | 50 |
51 |
52 |
53 |
54 | 55 |

Search

56 |
57 | 58 |

59 | Please activate JavaScript to enable the search 60 | functionality. 61 |

62 |
63 |

64 | From here you can search these documents. Enter your search 65 | words into the box below and click "search". Note that the search 66 | function will automatically search for all of the words. Pages 67 | containing fewer words won't appear in the result list. 68 |

69 |
70 | 71 | 72 | 73 |
74 | 75 |
76 | 77 |
78 | 79 |
80 |
81 |
82 |
83 |
84 |
85 |
86 |
87 |
88 |
89 |

Navigation

90 | 99 |
100 |
101 | © Copyright 2014, Timothy Ho, Nicholas Bacuez. 102 | Created using Sphinx 1.2.2. 103 |
104 | 105 | -------------------------------------------------------------------------------- /docs/_build/html/searchindex.js: -------------------------------------------------------------------------------- 1 | Search.setIndex({envversion:42,terms:{unari:3,represent:2,all:[1,2,3],text:3,random:4,oper:3,disj:[2,4],foundoptionalgroup:3,processopt:[2,4],find:[2,3,4],paramet:[1,2,3,4],whose:1,group:[2,3,4],advantag:4,chosen:4,lai:1,deterministicgener:[2,3],add:1,skip:4,main:3,take:[2,4],opt:[2,4],foundweightedexpress:3,non:1,independ:3,"return":[1,2,3,4],string:[0,1,2,3,4],thei:4,get:[1,2,3],python:[2,3,4],sentenc:4,processnontermin:[2,4],number:[0,3,4],getgrammarobject:3,should:[2,3],nontermin:[1,2,3,4],recurs:[0,2],requir:[2,4],like:4,portion:2,grammar:[],name:[1,3],weightedchoic:4,list:[1,2,3,4],separ:4,provid:4,token:3,gram:[2,3,4],each:[2,4],found:3,higher:4,listofset:[2,4],side:[1,3],depend:2,exceed:2,set:[0,2,3,4],deal:2,some:0,idea:[3,4],result:[2,3,4],run:[2,3,4],correspond:2,out:1,variabl:1,index:0,disjunct:[1,3,4],total:0,bottom:3,definit:[1,3],label:1,content:[0,3],expans:[1,2,3,4],print:3,"import":[2,3],notabl:3,"public":[1,2,3,4],refer:[2,3],altern:[2,3,4],themselv:2,loc:3,manipul:0,entir:3,getrh:1,path:[2,4],processsequ:[2,4],tok:3,addit:0,search:0,pypars:3,ideasnonrecurs:2,addrul:1,processrh:[2,4],script:3,equal:4,page:0,where:[2,4],base:4,action:3,produc:3,chanc:4,first:4,comment:3,semant:3,simpli:1,directli:2,point:3,nocom:3,"float":[3,4],ntname:1,two:[0,4],mai:2,addpublicrul:1,empti:2,strip:3,artifici:0,foundtoken:3,jsgf:[],from:[2,3,4],seq:[2,4],foundweight:3,compos:3,been:3,otherwis:2,accumul:3,adjac:[2,4],call:[2,3],includ:[0,2,3,4],statement:3,handl:3,termin:[1,2],type:2,store:1,more:3,"function":[2,3],option:[1,2,3,4],tupl:[2,4],weight:[2,3,4],pars:3,line:3,origin:3,concaten:[2,3,4],possibl:1,whether:2,foundseq:3,remov:3,corpora:0,maximum:2,oldlin:3,defin:[2,3,4],calcul:0,can:[0,2,3,4],fault:2,aid:0,foundnontermin:3,probabilisticgener:[3,4],featur:3,file:[1,2,3,4],creat:[0,3],process:4,jsgfgrammar:3,parser:[0,3],argument:[2,3,4],repres:[1,3],tag:3,right:[1,3],have:[3,4],rulenam:3,combineset:[2,4],check:0,greater:4,foundpair:3,listoftupl:4,stream:3,jsgfexpress:1,when:3,cross:[2,4],same:3,filestream:3,other:4,futur:0,which:[0,1],you:[2,4],probabl:4,either:[1,2,4],product:[2,4],express:[2,3,4],intend:0,structur:1,sequenc:[2,3,4],object:[3,4],upon:3,hand:[1,3],processdisjunct:[2,4],regular:3,probabilist:[],pair:3,build:3,contain:[1,3],segment:2,"class":1,expand:4,choos:4,indirectli:2,rule:[1,2,3,4],assign:4,pipe:4,depth:2,without:4,combin:[2,4],error:2,thi:[0,1,2,3,4],time:4,far:3,element:[1,2,3,4],order:3,left:1},objtypes:{"0":"py:module","1":"py:class","2":"py:function","3":"py:method"},objnames:{"0":["py","module","Python module"],"1":["py","class","Python class"],"2":["py","function","Python function"],"3":["py","method","Python method"]},filenames:["index","JSGFGrammar","DeterministicGenerator","JSGFParser","ProbabilisticGenerator"],titles:["Welcome to JSGF Grammar Tools’s documentation!","JSGFGrammar module","Deterministic Generator module","JSGFParser module","Probabilistic Generator module"],objects:{"":{JSGFGrammar:[1,0,0,"-"],DeterministicGenerator:[2,0,0,"-"],JSGFParser:[3,0,0,"-"],ProbabilisticGenerator:[4,0,0,"-"]},DeterministicGenerator:{processDisjunction:[2,2,1,""],processOptional:[2,2,1,""],processRHS:[2,2,1,""],processNonTerminal:[2,2,1,""],combineSets:[2,2,1,""],processSequence:[2,2,1,""]},JSGFGrammar:{Grammar:[1,1,1,""],Rule:[1,1,1,""],JSGFExpression:[1,1,1,""],Disjunction:[1,1,1,""],Optional:[1,1,1,""],NonTerminal:[1,1,1,""]},"JSGFGrammar.Grammar":{addRule:[1,3,1,""],getRHS:[1,3,1,""],addPublicRule:[1,3,1,""]},JSGFParser:{nocomment:[3,2,1,""],foundWeight:[3,2,1,""],foundSeq:[3,2,1,""],foundToken:[3,2,1,""],foundPair:[3,2,1,""],getGrammarObject:[3,2,1,""],foundNonterminal:[3,2,1,""],foundOptionalGroup:[3,2,1,""],foundWeightedExpression:[3,2,1,""]},ProbabilisticGenerator:{processDisjunction:[4,2,1,""],processOptional:[4,2,1,""],processRHS:[4,2,1,""],processNonTerminal:[4,2,1,""],combineSets:[4,2,1,""],weightedChoice:[4,2,1,""],processSequence:[4,2,1,""]}},titleterms:{determinist:2,jsgf:0,grammar:0,welcom:0,gener:[2,4],jsgfgrammar:1,probabilist:4,jsgfparser:3,indic:0,tabl:0,modul:[1,2,3,4],document:0,tool:0}}) -------------------------------------------------------------------------------- /docs/_build/html/_static/pygments.css: -------------------------------------------------------------------------------- 1 | .highlight .hll { background-color: #ffffcc } 2 | .highlight { background: #eeffcc; } 3 | .highlight .c { color: #408090; font-style: italic } /* Comment */ 4 | .highlight .err { border: 1px solid #FF0000 } /* Error */ 5 | .highlight .k { color: #007020; font-weight: bold } /* Keyword */ 6 | .highlight .o { color: #666666 } /* Operator */ 7 | .highlight .cm { color: #408090; font-style: italic } /* Comment.Multiline */ 8 | .highlight .cp { color: #007020 } /* Comment.Preproc */ 9 | .highlight .c1 { color: #408090; font-style: italic } /* Comment.Single */ 10 | .highlight .cs { color: #408090; background-color: #fff0f0 } /* Comment.Special */ 11 | .highlight .gd { color: #A00000 } /* Generic.Deleted */ 12 | .highlight .ge { font-style: italic } /* Generic.Emph */ 13 | .highlight .gr { color: #FF0000 } /* Generic.Error */ 14 | .highlight .gh { color: #000080; font-weight: bold } /* Generic.Heading */ 15 | .highlight .gi { color: #00A000 } /* Generic.Inserted */ 16 | .highlight .go { color: #333333 } /* Generic.Output */ 17 | .highlight .gp { color: #c65d09; font-weight: bold } /* Generic.Prompt */ 18 | .highlight .gs { font-weight: bold } /* Generic.Strong */ 19 | .highlight .gu { color: #800080; font-weight: bold } /* Generic.Subheading */ 20 | .highlight .gt { color: #0044DD } /* Generic.Traceback */ 21 | .highlight .kc { color: #007020; font-weight: bold } /* Keyword.Constant */ 22 | .highlight .kd { color: #007020; font-weight: bold } /* Keyword.Declaration */ 23 | .highlight .kn { color: #007020; font-weight: bold } /* Keyword.Namespace */ 24 | .highlight .kp { color: #007020 } /* Keyword.Pseudo */ 25 | .highlight .kr { color: #007020; font-weight: bold } /* Keyword.Reserved */ 26 | .highlight .kt { color: #902000 } /* Keyword.Type */ 27 | .highlight .m { color: #208050 } /* Literal.Number */ 28 | .highlight .s { color: #4070a0 } /* Literal.String */ 29 | .highlight .na { color: #4070a0 } /* Name.Attribute */ 30 | .highlight .nb { color: #007020 } /* Name.Builtin */ 31 | .highlight .nc { color: #0e84b5; font-weight: bold } /* Name.Class */ 32 | .highlight .no { color: #60add5 } /* Name.Constant */ 33 | .highlight .nd { color: #555555; font-weight: bold } /* Name.Decorator */ 34 | .highlight .ni { color: #d55537; font-weight: bold } /* Name.Entity */ 35 | .highlight .ne { color: #007020 } /* Name.Exception */ 36 | .highlight .nf { color: #06287e } /* Name.Function */ 37 | .highlight .nl { color: #002070; font-weight: bold } /* Name.Label */ 38 | .highlight .nn { color: #0e84b5; font-weight: bold } /* Name.Namespace */ 39 | .highlight .nt { color: #062873; font-weight: bold } /* Name.Tag */ 40 | .highlight .nv { color: #bb60d5 } /* Name.Variable */ 41 | .highlight .ow { color: #007020; font-weight: bold } /* Operator.Word */ 42 | .highlight .w { color: #bbbbbb } /* Text.Whitespace */ 43 | .highlight .mf { color: #208050 } /* Literal.Number.Float */ 44 | .highlight .mh { color: #208050 } /* Literal.Number.Hex */ 45 | .highlight .mi { color: #208050 } /* Literal.Number.Integer */ 46 | .highlight .mo { color: #208050 } /* Literal.Number.Oct */ 47 | .highlight .sb { color: #4070a0 } /* Literal.String.Backtick */ 48 | .highlight .sc { color: #4070a0 } /* Literal.String.Char */ 49 | .highlight .sd { color: #4070a0; font-style: italic } /* Literal.String.Doc */ 50 | .highlight .s2 { color: #4070a0 } /* Literal.String.Double */ 51 | .highlight .se { color: #4070a0; font-weight: bold } /* Literal.String.Escape */ 52 | .highlight .sh { color: #4070a0 } /* Literal.String.Heredoc */ 53 | .highlight .si { color: #70a0d0; font-style: italic } /* Literal.String.Interpol */ 54 | .highlight .sx { color: #c65d09 } /* Literal.String.Other */ 55 | .highlight .sr { color: #235388 } /* Literal.String.Regex */ 56 | .highlight .s1 { color: #4070a0 } /* Literal.String.Single */ 57 | .highlight .ss { color: #517918 } /* Literal.String.Symbol */ 58 | .highlight .bp { color: #007020 } /* Name.Builtin.Pseudo */ 59 | .highlight .vc { color: #bb60d5 } /* Name.Variable.Class */ 60 | .highlight .vg { color: #bb60d5 } /* Name.Variable.Global */ 61 | .highlight .vi { color: #bb60d5 } /* Name.Variable.Instance */ 62 | .highlight .il { color: #208050 } /* Literal.Number.Integer.Long */ -------------------------------------------------------------------------------- /jsgf/ast_nodes.py: -------------------------------------------------------------------------------- 1 | """ 2 | Abstract Syntax Tree nodes for JSGF grammars. 3 | 4 | This module provides the core AST node classes that represent different 5 | parts of a JSGF grammar structure. 6 | """ 7 | 8 | from typing import List, Union, Any, Optional 9 | from abc import ABC, abstractmethod 10 | 11 | 12 | class JSGFNode(ABC): 13 | """Base class for all JSGF AST nodes.""" 14 | 15 | @abstractmethod 16 | def __str__(self) -> str: 17 | """Return a string representation of this node.""" 18 | pass 19 | 20 | def __repr__(self) -> str: 21 | return f"{self.__class__.__name__}({str(self)})" 22 | 23 | 24 | class Terminal(JSGFNode): 25 | """Represents a terminal symbol (token) in the grammar.""" 26 | 27 | def __init__(self, value: str): 28 | self.value = value 29 | 30 | def __str__(self) -> str: 31 | return self.value 32 | 33 | def __eq__(self, other: Any) -> bool: 34 | return isinstance(other, Terminal) and self.value == other.value 35 | 36 | def __hash__(self) -> int: 37 | return hash(self.value) 38 | 39 | 40 | class NonTerminal(JSGFNode): 41 | """Represents a non-terminal symbol in the grammar.""" 42 | 43 | def __init__(self, name: str): 44 | self.name = name 45 | 46 | def __str__(self) -> str: 47 | return self.name 48 | 49 | def __eq__(self, other: Any) -> bool: 50 | return isinstance(other, NonTerminal) and self.name == other.name 51 | 52 | def __hash__(self) -> int: 53 | return hash(self.name) 54 | 55 | 56 | class Sequence(JSGFNode): 57 | """Represents a sequence of elements.""" 58 | 59 | def __init__(self, elements: List[JSGFNode]): 60 | self.elements = elements 61 | 62 | def __str__(self) -> str: 63 | return " ".join(str(element) for element in self.elements) 64 | 65 | def __iter__(self): 66 | return iter(self.elements) 67 | 68 | def __len__(self) -> int: 69 | return len(self.elements) 70 | 71 | def __getitem__(self, index: int) -> JSGFNode: 72 | return self.elements[index] 73 | 74 | 75 | class Alternative(JSGFNode): 76 | """Represents alternatives (choices) in the grammar.""" 77 | 78 | def __init__(self, choices: List[Union[JSGFNode, tuple]]): 79 | """ 80 | Initialize alternatives. 81 | 82 | Args: 83 | choices: List of choices. Each choice can be: 84 | - A JSGFNode (unweighted) 85 | - A tuple of (JSGFNode, weight) (weighted) 86 | """ 87 | self.choices = [] 88 | for choice in choices: 89 | if isinstance(choice, tuple): 90 | node, weight = choice 91 | self.choices.append((node, float(weight))) 92 | else: 93 | self.choices.append((choice, 1.0)) # Default weight 94 | 95 | def __str__(self) -> str: 96 | choice_strs = [] 97 | for node, weight in self.choices: 98 | if weight != 1.0: 99 | choice_strs.append(f"/{weight}/ {node}") 100 | else: 101 | choice_strs.append(str(node)) 102 | return "( " + " | ".join(choice_strs) + " )" 103 | 104 | def __iter__(self): 105 | return iter(self.choices) 106 | 107 | def __len__(self) -> int: 108 | return len(self.choices) 109 | 110 | def get_weights(self) -> List[float]: 111 | """Return the weights of all choices.""" 112 | return [weight for _, weight in self.choices] 113 | 114 | def get_nodes(self) -> List[JSGFNode]: 115 | """Return the nodes of all choices.""" 116 | return [node for node, _ in self.choices] 117 | 118 | 119 | class Optional(JSGFNode): 120 | """Represents an optional element in the grammar.""" 121 | 122 | def __init__(self, element: JSGFNode): 123 | self.element = element 124 | 125 | def __str__(self) -> str: 126 | return f"[ {self.element} ]" 127 | 128 | 129 | class Group(JSGFNode): 130 | """Represents a grouped element.""" 131 | 132 | def __init__(self, element: JSGFNode): 133 | self.element = element 134 | 135 | def __str__(self) -> str: 136 | return f"( {self.element} )" 137 | 138 | 139 | class Rule: 140 | """Represents a complete grammar rule.""" 141 | 142 | def __init__(self, name: str, expansion: JSGFNode, is_public: bool = False): 143 | self.name = name 144 | self.expansion = expansion 145 | self.is_public = is_public 146 | 147 | def __str__(self) -> str: 148 | prefix = "public " if self.is_public else "" 149 | return f"{prefix}<{self.name}> = {self.expansion};" 150 | 151 | def __repr__(self) -> str: 152 | return f"Rule(name='{self.name}', is_public={self.is_public})" -------------------------------------------------------------------------------- /docs/_build/html/py-modindex.html: -------------------------------------------------------------------------------- 1 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | Python Module Index — JSGF Grammar Tools 1.0 documentation 10 | 11 | 12 | 13 | 14 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 32 | 33 | 34 | 35 | 36 |
37 |

Navigation

38 | 47 |
48 | 49 |
50 |
51 |
52 |
53 | 54 | 55 |

Python Module Index

56 | 57 |
58 | d | 59 | j | 60 | p 61 |
62 | 63 | 64 | 65 | 67 | 68 | 69 | 72 | 73 | 75 | 76 | 77 | 80 | 81 | 82 | 85 | 86 | 88 | 89 | 90 | 93 |
 
66 | d
70 | DeterministicGenerator 71 |
 
74 | j
78 | JSGFGrammar 79 |
83 | JSGFParser 84 |
 
87 | p
91 | ProbabilisticGenerator 92 |
94 | 95 | 96 |
97 |
98 |
99 |
100 |
101 | 113 | 114 |
115 |
116 |
117 |
118 |
119 |

Navigation

120 | 129 |
130 |
131 | © Copyright 2014, Timothy Ho, Nicholas Bacuez. 132 | Created using Sphinx 1.2.2. 133 |
134 | 135 | -------------------------------------------------------------------------------- /docs/_build/html/_static/default.css: -------------------------------------------------------------------------------- 1 | /* 2 | * default.css_t 3 | * ~~~~~~~~~~~~~ 4 | * 5 | * Sphinx stylesheet -- default theme. 6 | * 7 | * :copyright: Copyright 2007-2014 by the Sphinx team, see AUTHORS. 8 | * :license: BSD, see LICENSE for details. 9 | * 10 | */ 11 | 12 | @import url("basic.css"); 13 | 14 | /* -- page layout ----------------------------------------------------------- */ 15 | 16 | body { 17 | font-family: sans-serif; 18 | font-size: 100%; 19 | background-color: #11303d; 20 | color: #000; 21 | margin: 0; 22 | padding: 0; 23 | } 24 | 25 | div.document { 26 | background-color: #1c4e63; 27 | } 28 | 29 | div.documentwrapper { 30 | float: left; 31 | width: 100%; 32 | } 33 | 34 | div.bodywrapper { 35 | margin: 0 0 0 230px; 36 | } 37 | 38 | div.body { 39 | background-color: #ffffff; 40 | color: #000000; 41 | padding: 0 20px 30px 20px; 42 | } 43 | 44 | div.footer { 45 | color: #ffffff; 46 | width: 100%; 47 | padding: 9px 0 9px 0; 48 | text-align: center; 49 | font-size: 75%; 50 | } 51 | 52 | div.footer a { 53 | color: #ffffff; 54 | text-decoration: underline; 55 | } 56 | 57 | div.related { 58 | background-color: #133f52; 59 | line-height: 30px; 60 | color: #ffffff; 61 | } 62 | 63 | div.related a { 64 | color: #ffffff; 65 | } 66 | 67 | div.sphinxsidebar { 68 | } 69 | 70 | div.sphinxsidebar h3 { 71 | font-family: 'Trebuchet MS', sans-serif; 72 | color: #ffffff; 73 | font-size: 1.4em; 74 | font-weight: normal; 75 | margin: 0; 76 | padding: 0; 77 | } 78 | 79 | div.sphinxsidebar h3 a { 80 | color: #ffffff; 81 | } 82 | 83 | div.sphinxsidebar h4 { 84 | font-family: 'Trebuchet MS', sans-serif; 85 | color: #ffffff; 86 | font-size: 1.3em; 87 | font-weight: normal; 88 | margin: 5px 0 0 0; 89 | padding: 0; 90 | } 91 | 92 | div.sphinxsidebar p { 93 | color: #ffffff; 94 | } 95 | 96 | div.sphinxsidebar p.topless { 97 | margin: 5px 10px 10px 10px; 98 | } 99 | 100 | div.sphinxsidebar ul { 101 | margin: 10px; 102 | padding: 0; 103 | color: #ffffff; 104 | } 105 | 106 | div.sphinxsidebar a { 107 | color: #98dbcc; 108 | } 109 | 110 | div.sphinxsidebar input { 111 | border: 1px solid #98dbcc; 112 | font-family: sans-serif; 113 | font-size: 1em; 114 | } 115 | 116 | 117 | 118 | /* -- hyperlink styles ------------------------------------------------------ */ 119 | 120 | a { 121 | color: #355f7c; 122 | text-decoration: none; 123 | } 124 | 125 | a:visited { 126 | color: #355f7c; 127 | text-decoration: none; 128 | } 129 | 130 | a:hover { 131 | text-decoration: underline; 132 | } 133 | 134 | 135 | 136 | /* -- body styles ----------------------------------------------------------- */ 137 | 138 | div.body h1, 139 | div.body h2, 140 | div.body h3, 141 | div.body h4, 142 | div.body h5, 143 | div.body h6 { 144 | font-family: 'Trebuchet MS', sans-serif; 145 | background-color: #f2f2f2; 146 | font-weight: normal; 147 | color: #20435c; 148 | border-bottom: 1px solid #ccc; 149 | margin: 20px -20px 10px -20px; 150 | padding: 3px 0 3px 10px; 151 | } 152 | 153 | div.body h1 { margin-top: 0; font-size: 200%; } 154 | div.body h2 { font-size: 160%; } 155 | div.body h3 { font-size: 140%; } 156 | div.body h4 { font-size: 120%; } 157 | div.body h5 { font-size: 110%; } 158 | div.body h6 { font-size: 100%; } 159 | 160 | a.headerlink { 161 | color: #c60f0f; 162 | font-size: 0.8em; 163 | padding: 0 4px 0 4px; 164 | text-decoration: none; 165 | } 166 | 167 | a.headerlink:hover { 168 | background-color: #c60f0f; 169 | color: white; 170 | } 171 | 172 | div.body p, div.body dd, div.body li { 173 | text-align: justify; 174 | line-height: 130%; 175 | } 176 | 177 | div.admonition p.admonition-title + p { 178 | display: inline; 179 | } 180 | 181 | div.admonition p { 182 | margin-bottom: 5px; 183 | } 184 | 185 | div.admonition pre { 186 | margin-bottom: 5px; 187 | } 188 | 189 | div.admonition ul, div.admonition ol { 190 | margin-bottom: 5px; 191 | } 192 | 193 | div.note { 194 | background-color: #eee; 195 | border: 1px solid #ccc; 196 | } 197 | 198 | div.seealso { 199 | background-color: #ffc; 200 | border: 1px solid #ff6; 201 | } 202 | 203 | div.topic { 204 | background-color: #eee; 205 | } 206 | 207 | div.warning { 208 | background-color: #ffe4e4; 209 | border: 1px solid #f66; 210 | } 211 | 212 | p.admonition-title { 213 | display: inline; 214 | } 215 | 216 | p.admonition-title:after { 217 | content: ":"; 218 | } 219 | 220 | pre { 221 | padding: 5px; 222 | background-color: #eeffcc; 223 | color: #333333; 224 | line-height: 120%; 225 | border: 1px solid #ac9; 226 | border-left: none; 227 | border-right: none; 228 | } 229 | 230 | tt { 231 | background-color: #ecf0f3; 232 | padding: 0 1px 0 1px; 233 | font-size: 0.95em; 234 | } 235 | 236 | th { 237 | background-color: #ede; 238 | } 239 | 240 | .warning tt { 241 | background: #efc2c2; 242 | } 243 | 244 | .note tt { 245 | background: #d6d6d6; 246 | } 247 | 248 | .viewcode-back { 249 | font-family: sans-serif; 250 | } 251 | 252 | div.viewcode-block:target { 253 | background-color: #f4debf; 254 | border-top: 1px solid #ac9; 255 | border-bottom: 1px solid #ac9; 256 | } -------------------------------------------------------------------------------- /JSGFGrammar.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # @copyright: MIT License 3 | # Copyright (c) 2018 syntactic (Pastèque Ho) 4 | # Permission is hereby granted, free of charge, to any person obtaining a copy 5 | # of this software and associated documentation files (the "Software"), to deal 6 | # in the Software without restriction, including without limitation the rights 7 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 8 | # copies of the Software, and to permit persons to whom the Software is 9 | # furnished to do so, subject to the following conditions: 10 | # The above copyright notice and this permission notice shall be included in all 11 | # copies or substantial portions of the Software. 12 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 13 | # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 14 | # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 15 | # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 16 | # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 17 | # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 18 | # SOFTWARE. 19 | # @summary: This file lays out the class structure for a JSGF Grammar 20 | # @since: 2014/06/02 21 | 22 | """ 23 | This file lays out the class structure for a JSGF Grammar. 24 | 25 | .. module:: JSGFGrammar 26 | 27 | .. moduleauthor:: Pastèque Ho 28 | """ 29 | 30 | 31 | class JSGFExpression(): 32 | pass 33 | 34 | class Disjunction(JSGFExpression): 35 | """ 36 | Disjunction class stores disjuncts in a list 37 | """ 38 | 39 | def __init__(self, disjuncts): 40 | self.disjuncts = disjuncts 41 | 42 | def __str__(self): 43 | return '( ' + ' | '.join(map(str,self.disjuncts)) + ' )' 44 | 45 | def __repr__(self): 46 | return str(self) 47 | 48 | def __getitem__(self): 49 | return self.disjuncts 50 | 51 | class Optional(JSGFExpression): 52 | """ 53 | Optional class stores either a JSGFExpression, list, or string 54 | as its optional element 55 | """ 56 | 57 | def __init__(self, option): 58 | self.option = option 59 | 60 | def __str__(self): 61 | return ('[ ' + str(self.option) + ' ]') 62 | 63 | def __repr__(self): 64 | return str(self) 65 | 66 | def __getitem__(self): 67 | return self.option 68 | 69 | class NonTerminal(JSGFExpression): 70 | """ 71 | NonTerminal class simply stores the label of the nonterminal 72 | """ 73 | 74 | def __init__(self, ntName): 75 | self.name = ntName 76 | 77 | def __str__(self): 78 | return self.name 79 | 80 | def __repr__(self): 81 | return str(self) 82 | 83 | def __getitem__(self): 84 | return self.name 85 | 86 | 87 | class Rule(): 88 | """ 89 | Rule class, represents a JSGF rule, with a nonterminal name representing the 90 | left hand side, and a list of possible expansions representing the right 91 | hand side. 92 | """ 93 | 94 | def __init__(self): 95 | """ 96 | constructor with no args 97 | """ 98 | self.lhs = '' 99 | self.rhs = [] 100 | 101 | def __init__(self, lhs): 102 | """ 103 | constructor with nonterminal name 104 | """ 105 | pass 106 | 107 | def __init__(self, lhs, rhs): 108 | """ 109 | constructor with full rule definition 110 | """ 111 | self.lhs = lhs 112 | self.rhs = rhs 113 | 114 | def __str__(self): 115 | return (str(self.lhs) + ' -> ' + str(self.rhs)) 116 | 117 | def __repr__(self): 118 | return str(self) 119 | 120 | 121 | class Grammar(): 122 | """ 123 | Grammar class which contains a list for public rules and a list 124 | for all rules. 125 | """ 126 | 127 | def __init__(self): 128 | self.rules = [] 129 | self.publicRules = [] 130 | 131 | def addRule(self, rule): 132 | """ 133 | adds a rule to the list of rules 134 | """ 135 | self.rules.append(rule) 136 | 137 | def addPublicRule(self, rule): 138 | """ 139 | adds a rule to the list of public rules 140 | """ 141 | self.publicRules.append(rule) 142 | 143 | def getRHS(self, nt): 144 | """ 145 | returns rule definition 146 | 147 | :param nt: Non-Terminal (variable) whose definition to get 148 | """ 149 | for rule in self.rules: 150 | #print 'checking', nt.name, 'and', rule.lhs, type(nt.name), rule.lhs.name 151 | if rule.lhs.name == nt.name: 152 | return rule.rhs 153 | raise ValueError("Rule not defined for " + str(nt)) 154 | 155 | def __getitem__(self, nt): 156 | #rhsides = [] to do for multiple rhsides 157 | for rule in self.rules: 158 | if rule.lhs == nt: 159 | return rule.rhs 160 | else: 161 | raise ValueError('Rule not defined for ' + str(nt)) 162 | 163 | def __str__(self): 164 | return 'All Rules:' + str(self.rules) + '\n' + 'Public Rules:' + str(self.publicRules) 165 | 166 | if __name__ == "__main__": 167 | jgDisj = Disjunction(['hello', 'world']) 168 | jgOpt = Optional(jgDisj) 169 | jgRule = Rule("", jgOpt) 170 | 171 | print(jgRule) 172 | -------------------------------------------------------------------------------- /docs/_build/html/_static/sidebar.js: -------------------------------------------------------------------------------- 1 | /* 2 | * sidebar.js 3 | * ~~~~~~~~~~ 4 | * 5 | * This script makes the Sphinx sidebar collapsible. 6 | * 7 | * .sphinxsidebar contains .sphinxsidebarwrapper. This script adds 8 | * in .sphixsidebar, after .sphinxsidebarwrapper, the #sidebarbutton 9 | * used to collapse and expand the sidebar. 10 | * 11 | * When the sidebar is collapsed the .sphinxsidebarwrapper is hidden 12 | * and the width of the sidebar and the margin-left of the document 13 | * are decreased. When the sidebar is expanded the opposite happens. 14 | * This script saves a per-browser/per-session cookie used to 15 | * remember the position of the sidebar among the pages. 16 | * Once the browser is closed the cookie is deleted and the position 17 | * reset to the default (expanded). 18 | * 19 | * :copyright: Copyright 2007-2014 by the Sphinx team, see AUTHORS. 20 | * :license: BSD, see LICENSE for details. 21 | * 22 | */ 23 | 24 | $(function() { 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | // global elements used by the functions. 34 | // the 'sidebarbutton' element is defined as global after its 35 | // creation, in the add_sidebar_button function 36 | var bodywrapper = $('.bodywrapper'); 37 | var sidebar = $('.sphinxsidebar'); 38 | var sidebarwrapper = $('.sphinxsidebarwrapper'); 39 | 40 | // for some reason, the document has no sidebar; do not run into errors 41 | if (!sidebar.length) return; 42 | 43 | // original margin-left of the bodywrapper and width of the sidebar 44 | // with the sidebar expanded 45 | var bw_margin_expanded = bodywrapper.css('margin-left'); 46 | var ssb_width_expanded = sidebar.width(); 47 | 48 | // margin-left of the bodywrapper and width of the sidebar 49 | // with the sidebar collapsed 50 | var bw_margin_collapsed = '.8em'; 51 | var ssb_width_collapsed = '.8em'; 52 | 53 | // colors used by the current theme 54 | var dark_color = $('.related').css('background-color'); 55 | var light_color = $('.document').css('background-color'); 56 | 57 | function sidebar_is_collapsed() { 58 | return sidebarwrapper.is(':not(:visible)'); 59 | } 60 | 61 | function toggle_sidebar() { 62 | if (sidebar_is_collapsed()) 63 | expand_sidebar(); 64 | else 65 | collapse_sidebar(); 66 | } 67 | 68 | function collapse_sidebar() { 69 | sidebarwrapper.hide(); 70 | sidebar.css('width', ssb_width_collapsed); 71 | bodywrapper.css('margin-left', bw_margin_collapsed); 72 | sidebarbutton.css({ 73 | 'margin-left': '0', 74 | 'height': bodywrapper.height() 75 | }); 76 | sidebarbutton.find('span').text('»'); 77 | sidebarbutton.attr('title', _('Expand sidebar')); 78 | document.cookie = 'sidebar=collapsed'; 79 | } 80 | 81 | function expand_sidebar() { 82 | bodywrapper.css('margin-left', bw_margin_expanded); 83 | sidebar.css('width', ssb_width_expanded); 84 | sidebarwrapper.show(); 85 | sidebarbutton.css({ 86 | 'margin-left': ssb_width_expanded-12, 87 | 'height': bodywrapper.height() 88 | }); 89 | sidebarbutton.find('span').text('«'); 90 | sidebarbutton.attr('title', _('Collapse sidebar')); 91 | document.cookie = 'sidebar=expanded'; 92 | } 93 | 94 | function add_sidebar_button() { 95 | sidebarwrapper.css({ 96 | 'float': 'left', 97 | 'margin-right': '0', 98 | 'width': ssb_width_expanded - 28 99 | }); 100 | // create the button 101 | sidebar.append( 102 | '
«
' 103 | ); 104 | var sidebarbutton = $('#sidebarbutton'); 105 | light_color = sidebarbutton.css('background-color'); 106 | // find the height of the viewport to center the '<<' in the page 107 | var viewport_height; 108 | if (window.innerHeight) 109 | viewport_height = window.innerHeight; 110 | else 111 | viewport_height = $(window).height(); 112 | sidebarbutton.find('span').css({ 113 | 'display': 'block', 114 | 'margin-top': (viewport_height - sidebar.position().top - 20) / 2 115 | }); 116 | 117 | sidebarbutton.click(toggle_sidebar); 118 | sidebarbutton.attr('title', _('Collapse sidebar')); 119 | sidebarbutton.css({ 120 | 'color': '#FFFFFF', 121 | 'border-left': '1px solid ' + dark_color, 122 | 'font-size': '1.2em', 123 | 'cursor': 'pointer', 124 | 'height': bodywrapper.height(), 125 | 'padding-top': '1px', 126 | 'margin-left': ssb_width_expanded - 12 127 | }); 128 | 129 | sidebarbutton.hover( 130 | function () { 131 | $(this).css('background-color', dark_color); 132 | }, 133 | function () { 134 | $(this).css('background-color', light_color); 135 | } 136 | ); 137 | } 138 | 139 | function set_position_from_cookie() { 140 | if (!document.cookie) 141 | return; 142 | var items = document.cookie.split(';'); 143 | for(var k=0; k = hello | hi; 62 | public = world | there; 63 | public = ; 64 | """ 65 | 66 | with StringIO(grammar_text) as f: 67 | grammar = parser.getGrammarObject(f) 68 | 69 | # Generate all possibilities (deterministic) 70 | det_gen.grammar = grammar 71 | rule = grammar.publicRules[2] # rule 72 | all_strings = det_gen.processRHS(rule.rhs) 73 | print("All possible strings:", all_strings) 74 | 75 | # Generate random string (probabilistic) 76 | prob_gen.grammar = grammar 77 | random_string = prob_gen.processRHS(rule.rhs) 78 | print("Random string:", random_string) 79 | ``` 80 | 81 | ## Grammar Format 82 | 83 | JSGFTools supports most of the JSGF specification: 84 | 85 | ```jsgf 86 | // Comments are supported 87 | public = ; 88 | 89 | // Alternatives with optional weights 90 | = /5/ hello | /1/ hi | hey; 91 | 92 | // Optional elements 93 | = [ please ]; 94 | 95 | // Nonterminal references 96 | = world | there; 97 | 98 | // Recursive rules (use with ProbabilisticGenerator only) 99 | = base | more; 100 | ``` 101 | 102 | ### Supported Features 103 | - Rule definitions and nonterminal references 104 | - Alternatives (|) with optional weights (/weight/) 105 | - Optional elements ([...]) 106 | - Grouping with parentheses 107 | - Comments (// and /* */) 108 | - Public and private rules 109 | - **Unicode support** for 10+ major language scripts 110 | 111 | ### Unicode Support 112 | 113 | JSGFTools fully supports Unicode characters in both tokens and rule names, covering: 114 | - **Latin scripts** (English, Spanish, French, etc.) 115 | - **CJK** (Chinese, Japanese Kanji, Korean Hanja) 116 | - **Arabic** (Arabic, Persian, Urdu) 117 | - **Cyrillic** (Russian, Ukrainian, Bulgarian) 118 | - **Devanagari** (Hindi, Sanskrit, Marathi) 119 | - **Hangul** (Korean) 120 | - **Hebrew** 121 | - **Greek** 122 | - **Thai** 123 | 124 | Example: 125 | ```jsgf 126 | public = hello | 你好 | こんにちは | مرحبا | привет | שלום; 127 | public <问候> = 您好 | 欢迎; 128 | ``` 129 | 130 | ### Not Yet Supported 131 | - Kleene operators (* and +) 132 | - Import statements 133 | - Tags 134 | 135 | ## Important Notes 136 | 137 | ### Recursive vs Non-Recursive Grammars 138 | 139 | - **DeterministicGenerator**: Only use with non-recursive grammars to avoid infinite loops 140 | - **ProbabilisticGenerator**: Can safely handle recursive grammars through probabilistic termination 141 | 142 | **Example of recursive rule:** 143 | ```jsgf 144 | = | and ; 145 | ``` 146 | 147 | ## Testing 148 | 149 | Run the test suite: 150 | ```bash 151 | pytest test_jsgf_tools.py -v 152 | ``` 153 | 154 | Run specific test categories: 155 | ```bash 156 | pytest test_jsgf_tools.py::TestJSGFParser -v # Parser tests 157 | pytest test_jsgf_tools.py::TestIntegration -v # Integration tests 158 | ``` 159 | 160 | ## Documentation 161 | 162 | For detailed API documentation, build the Sphinx docs: 163 | ```bash 164 | cd docs 165 | make html 166 | ``` 167 | 168 | Then open `docs/_build/html/index.html` in your browser. 169 | 170 | ## Example Files 171 | 172 | - `Ideas.gram`: Recursive grammar example (use with ProbabilisticGenerator) 173 | - `IdeasNonRecursive.gram`: Non-recursive grammar example (use with DeterministicGenerator) 174 | 175 | ## Contributing 176 | 177 | 1. Fork the repository 178 | 2. Create a feature branch 179 | 3. Make your changes 180 | 4. Add tests for new functionality 181 | 5. Run the test suite: `pytest` 182 | 6. Submit a pull request 183 | 184 | ## License 185 | 186 | MIT License. See [LICENSE](LICENSE) file for details. 187 | 188 | ## Version History 189 | 190 | - **2.1.1**: Fixed argparse support in DeterministicGenerator CLI (--help now works) 191 | - **2.1.0**: Added comprehensive Unicode support (10+ language scripts), published to PyPI 192 | - **2.0.0**: Complete Python 3 modernization, added test suite, improved packaging 193 | - **1.x**: Original Python 2.7 version 194 | -------------------------------------------------------------------------------- /docs/_build/html/index.html: -------------------------------------------------------------------------------- 1 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | Welcome to JSGF Grammar Tools’s documentation! — JSGF Grammar Tools 1.0 documentation 10 | 11 | 12 | 13 | 14 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 |
31 |

Navigation

32 | 44 |
45 | 46 |
47 |
48 |
49 |
50 | 51 |
52 |

Welcome to JSGF Grammar Tools’s documentation!

53 |

This set of tools is intended to aid the manipulation of 54 | JSGF Grammars. It includes a parser, and two string generators which 55 | can be used to create artificial corpora. Some additional tools that will 56 | be added in the near future include a tool to check grammars for recursion 57 | and a tool to calculate the total number of strings a grammar generates.

58 |

Contents:

59 |
60 | 66 |
67 |
68 |
69 |

Indices and tables

70 | 75 |
76 | 77 | 78 |
79 |
80 |
81 |
82 |
83 |

Table Of Contents

84 | 88 | 89 |

Next topic

90 |

JSGFGrammar module

92 |

This Page

93 | 97 | 109 | 110 |
111 |
112 |
113 |
114 |
115 |

Navigation

116 | 128 |
129 |
130 | © Copyright 2014, Timothy Ho, Nicholas Bacuez. 131 | Created using Sphinx 1.2.2. 132 |
133 | 134 | -------------------------------------------------------------------------------- /DeterministicGenerator.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # @copyright: MIT License 3 | # Copyright (c) 2018 syntactic (Pastèque Ho) 4 | # Permission is hereby granted, free of charge, to any person obtaining a copy 5 | # of this software and associated documentation files (the "Software"), to deal 6 | # in the Software without restriction, including without limitation the rights 7 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 8 | # copies of the Software, and to permit persons to whom the Software is 9 | # furnished to do so, subject to the following conditions: 10 | # The above copyright notice and this permission notice shall be included in all 11 | # copies or substantial portions of the Software. 12 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 13 | # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 14 | # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 15 | # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 16 | # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 17 | # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 18 | # SOFTWARE. 19 | # @summary: This file generates all strings described by a non-recursive JSGF grammar. 20 | # Run it by entering into the command line: python DeterministicGenerator.py 21 | # where is the path to the JSGF grammar. 22 | # @since: 2014/06/02 23 | 24 | """ 25 | This file deterministically generates strings from a JSGF Grammar, whether there are \ 26 | weights defined in rules or not. It requires one argument: the path to the JSGF\ 27 | Grammar file. You can run this on the included grammar IdeasNonRecursive.gram:\ 28 | 29 | 30 | ``python DeterministicGenerator.py IdeasNonRecursive.gram`` 31 | 32 | This will generate all strings defined by the public rules of IdeasNonRecursive.gram.\ 33 | It is important that the grammar used by the generator is not recursive (rules \ 34 | should not directly or indirectly reference themselves), so that the generator\ 35 | terminates. Otherwise, you may get a maximum recursion depth exceeded error or \ 36 | a segmentation fault. 37 | """ 38 | 39 | import sys, itertools, argparse 40 | import JSGFParser as parser 41 | import JSGFGrammar as gram 42 | 43 | 44 | def combineSets(listOfSets): 45 | """ 46 | Combines sets of strings by taking the cross product of the sets and \ 47 | concatenating the elements in the resulting tuples 48 | 49 | :param listOfSets: 2-D list of strings 50 | :returns: a list of strings 51 | """ 52 | totalCrossProduct = [''] 53 | for i in range(len(listOfSets)): 54 | currentProduct = [] 55 | for crossProduct in itertools.product(totalCrossProduct, listOfSets[i]): 56 | currentProduct.append((crossProduct[0].strip() + ' ' + crossProduct[1].strip()).strip()) 57 | totalCrossProduct = currentProduct 58 | return totalCrossProduct 59 | 60 | def processSequence(seq): 61 | """ 62 | Combines adjacent elements in a sequence 63 | """ 64 | componentSets = [] 65 | for component in seq: 66 | componentSets.append(processRHS(component)) 67 | return combineSets(componentSets) 68 | 69 | 70 | def processNonTerminal(nt): 71 | """ 72 | Finds the rule expansion for a nonterminal and returns its expansion. 73 | """ 74 | return processRHS(grammar.getRHS(nt)) 75 | 76 | def processDisjunction(disj): 77 | """ 78 | Returns the string representations of a set of alternatives 79 | 80 | :returns: list of strings, where each string is each alternative 81 | """ 82 | disjunctExpansions = [] 83 | if type(disj.disjuncts[0]) is tuple: 84 | disjuncts = map(lambda x : x[0], disj.disjuncts) 85 | else: 86 | disjuncts = disj.disjuncts 87 | for disjunct in disjuncts: 88 | disjunctExpansions.extend(processRHS(disjunct)) 89 | return disjunctExpansions 90 | 91 | def processOptional(opt): 92 | """ 93 | Returns the string representations of an optional grouping 94 | 95 | :type opt: JSGFOptional 96 | :returns: list of strings, including an empty string 97 | """ 98 | optional = [''] 99 | optional.extend(processRHS(opt.option)) 100 | return optional 101 | 102 | def processRHS(rhs): 103 | """ 104 | Depending on the type of the argument, calls the corresponding 105 | function to deal with that type. 106 | 107 | :param rhs: portion of JSGF rule 108 | :type rhs: either a JSGF Expression, list, or string 109 | :returns: list of strings 110 | """ 111 | if type(rhs) is list: 112 | return processSequence(rhs) 113 | elif isinstance(rhs, gram.Disjunction): 114 | return processDisjunction(rhs) 115 | elif isinstance(rhs, gram.Optional): 116 | return processOptional(rhs) 117 | elif isinstance(rhs, gram.NonTerminal): 118 | return processNonTerminal(rhs) 119 | elif isinstance(rhs, str): 120 | return [rhs] 121 | 122 | 123 | def main(): 124 | """Main function for command line usage""" 125 | global grammar 126 | 127 | arg_parser = argparse.ArgumentParser( 128 | description='Generate all possible strings from a non-recursive JSGF grammar' 129 | ) 130 | arg_parser.add_argument( 131 | 'grammarFile', 132 | help='Path to the JSGF grammar file' 133 | ) 134 | args = arg_parser.parse_args() 135 | 136 | try: 137 | with open(args.grammarFile, 'r') as fileStream: 138 | grammar = parser.getGrammarObject(fileStream) 139 | 140 | for rule in grammar.publicRules: 141 | expansions = processRHS(rule.rhs) 142 | for expansion in expansions: 143 | print(expansion) 144 | except FileNotFoundError: 145 | print(f"Error: Grammar file '{args.grammarFile}' not found") 146 | sys.exit(1) 147 | except Exception as e: 148 | print(f"Error processing grammar: {e}") 149 | sys.exit(1) 150 | 151 | if __name__ == '__main__': 152 | main() 153 | 154 | 155 | 156 | -------------------------------------------------------------------------------- /ProbabilisticGenerator.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | #/usr/bin/python 3 | 4 | # @copyright: MIT License 5 | # Copyright (c) 2018 syntactic (Pastèque Ho) 6 | # Permission is hereby granted, free of charge, to any person obtaining a copy 7 | # of this software and associated documentation files (the "Software"), to deal 8 | # in the Software without restriction, including without limitation the rights 9 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 10 | # copies of the Software, and to permit persons to whom the Software is 11 | # furnished to do so, subject to the following conditions: 12 | # The above copyright notice and this permission notice shall be included in all 13 | # copies or substantial portions of the Software. 14 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 15 | # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 16 | # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 17 | # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 18 | # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 19 | # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 20 | # SOFTWARE. 21 | # @summary: This file generates sentences from a PCFG in JSGF. Run it by entering 22 | # in the command line: python ProbabilisticGenerator.py 23 | # where is the path of the JSGF file, and is the number 24 | # of strings you want to generate 25 | # @since: 2014/06/02 26 | 27 | """ 28 | This file probabilistically generates strings from a JSGF grammar. It takes advantage \ 29 | of weights assigned to alternatives (separated by pipes) by choosing to \ 30 | expand higher weighted alternatives with greater probability. For sets of \ 31 | alternatives without weights, each alternative is equally likely to be \ 32 | expanded. For optional groups, the elements in the group have a 50% chance \ 33 | of being expanded. 34 | 35 | It requires two arguments: the path to the JSGF\ 36 | Grammar file, and the number of strings to generate. You can run this on the \ 37 | included grammar Ideas.gram:\ 38 | 39 | 40 | ``python ProbabilisticGenerator.py Ideas.gram 20`` 41 | 42 | This will generate 20 sentences based on the public rule(s) in Ideas.gram, using the \ 43 | weights if they are provided. 44 | """ 45 | 46 | import sys, itertools, random, bisect, argparse 47 | import JSGFParser as parser 48 | import JSGFGrammar as gram 49 | 50 | 51 | def weightedChoice(listOfTuples): 52 | """ 53 | Chooses an element of a list based on its weight 54 | 55 | :param listOfTuples: a list of (element, weight) tuples, where the element can be a JSGF expression object, string, or list,\ 56 | and the weight is a float 57 | :returns: the first element of a chosen tuple 58 | """ 59 | def accum(listOfWeights): # support function for creating ranges for weights 60 | for i in range(len(listOfWeights)): 61 | if i > 0: 62 | listOfWeights[i] += listOfWeights[i-1] 63 | return listOfWeights 64 | choices, weights = zip(*listOfTuples) 65 | cumdist = accum(list(weights)) 66 | x = random.random() * cumdist[-1] 67 | return choices[bisect.bisect(cumdist, x)] 68 | 69 | def combineSets(listOfSets): 70 | """ 71 | Combines sets of strings by taking the cross product of the sets and \ 72 | concatenating the elements in the resulting tuples 73 | 74 | :param listOfSets: 2-D list of strings 75 | :returns: a list of strings 76 | """ 77 | totalCrossProduct = [''] 78 | for i in range(len(listOfSets)): 79 | currentProduct = [] 80 | for crossProduct in itertools.product(totalCrossProduct, listOfSets[i]): 81 | #print crossProduct[0], crossProduct[1] 82 | currentProduct.append((crossProduct[0].strip() + ' ' + crossProduct[1].strip()).strip()) 83 | totalCrossProduct = currentProduct 84 | return totalCrossProduct 85 | 86 | def processSequence(seq): 87 | """ 88 | Combines adjacent elements in a sequence. 89 | """ 90 | componentSets = [] 91 | for component in seq: 92 | expandedComponent = processRHS(component).strip() 93 | if len(expandedComponent) > 0: 94 | componentSets.append(expandedComponent) 95 | return ' '.join(componentSets) 96 | 97 | 98 | def processNonTerminal(nt): 99 | """ 100 | Finds the rule expansion for a nonterminal and returns its expansion. 101 | """ 102 | return processRHS(grammar.getRHS(nt)) 103 | 104 | def processDisjunction(disj): 105 | """ 106 | Chooses either a random disjunct (for alternatives without weights) or 107 | a disjunct based on defined weights. 108 | """ 109 | if type(disj.disjuncts[0]) is tuple: 110 | return processRHS(weightedChoice(disj.disjuncts)) 111 | else: 112 | return processRHS(random.choice(disj.disjuncts)) 113 | 114 | def processOptional(opt): 115 | """ 116 | Processes the optional element 50% of the time, skips it the other 50% of the time 117 | """ 118 | rand = random.random() 119 | if rand <= 0.5: 120 | return '' 121 | else: 122 | return processRHS(opt.option) 123 | 124 | def processRHS(rhs): 125 | if type(rhs) is list: 126 | return processSequence(rhs) 127 | elif isinstance(rhs, gram.Disjunction): 128 | return processDisjunction(rhs) 129 | elif isinstance(rhs, gram.Optional): 130 | return processOptional(rhs) 131 | elif isinstance(rhs, gram.NonTerminal): 132 | return processNonTerminal(rhs) 133 | elif isinstance(rhs, str): 134 | return rhs 135 | 136 | 137 | def main(): 138 | """Main function for command line usage""" 139 | global grammar 140 | 141 | argParser = argparse.ArgumentParser(description='Generate random strings from a JSGF grammar') 142 | argParser.add_argument('grammarFile', help='Path to the JSGF grammar file') 143 | argParser.add_argument('iterations', type=int, help='Number of strings to generate') 144 | 145 | try: 146 | args = argParser.parse_args() 147 | except SystemExit: 148 | return 149 | 150 | try: 151 | with open(args.grammarFile, 'r') as fileStream: 152 | grammar = parser.getGrammarObject(fileStream) 153 | 154 | if len(grammar.publicRules) > 1: 155 | # Multiple public rules - create a disjunction of all of them 156 | disjuncts = [rule.rhs for rule in grammar.publicRules] 157 | newStartSymbol = gram.Disjunction(disjuncts) 158 | for i in range(args.iterations): 159 | print(processRHS(newStartSymbol)) 160 | else: 161 | # Single public rule 162 | startSymbol = grammar.publicRules[0] 163 | for i in range(args.iterations): 164 | expansions = processRHS(startSymbol.rhs) 165 | print(expansions) 166 | except FileNotFoundError: 167 | print(f"Error: Grammar file '{args.grammarFile}' not found") 168 | sys.exit(1) 169 | except Exception as e: 170 | print(f"Error processing grammar: {e}") 171 | sys.exit(1) 172 | 173 | if __name__ == '__main__': 174 | main() 175 | 176 | 177 | 178 | -------------------------------------------------------------------------------- /docs/Makefile: -------------------------------------------------------------------------------- 1 | # Makefile for Sphinx documentation 2 | # 3 | 4 | # You can set these variables from the command line. 5 | SPHINXOPTS = 6 | SPHINXBUILD = sphinx-build 7 | PAPER = 8 | BUILDDIR = _build 9 | 10 | # User-friendly check for sphinx-build 11 | ifeq ($(shell which $(SPHINXBUILD) >/dev/null 2>&1; echo $$?), 1) 12 | $(error The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '$(SPHINXBUILD)' executable. Alternatively you can add the directory with the executable to your PATH. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/) 13 | endif 14 | 15 | # Internal variables. 16 | PAPEROPT_a4 = -D latex_paper_size=a4 17 | PAPEROPT_letter = -D latex_paper_size=letter 18 | ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . 19 | # the i18n builder cannot share the environment and doctrees with the others 20 | I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . 21 | 22 | .PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest gettext 23 | 24 | help: 25 | @echo "Please use \`make ' where is one of" 26 | @echo " html to make standalone HTML files" 27 | @echo " dirhtml to make HTML files named index.html in directories" 28 | @echo " singlehtml to make a single large HTML file" 29 | @echo " pickle to make pickle files" 30 | @echo " json to make JSON files" 31 | @echo " htmlhelp to make HTML files and a HTML help project" 32 | @echo " qthelp to make HTML files and a qthelp project" 33 | @echo " devhelp to make HTML files and a Devhelp project" 34 | @echo " epub to make an epub" 35 | @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" 36 | @echo " latexpdf to make LaTeX files and run them through pdflatex" 37 | @echo " latexpdfja to make LaTeX files and run them through platex/dvipdfmx" 38 | @echo " text to make text files" 39 | @echo " man to make manual pages" 40 | @echo " texinfo to make Texinfo files" 41 | @echo " info to make Texinfo files and run them through makeinfo" 42 | @echo " gettext to make PO message catalogs" 43 | @echo " changes to make an overview of all changed/added/deprecated items" 44 | @echo " xml to make Docutils-native XML files" 45 | @echo " pseudoxml to make pseudoxml-XML files for display purposes" 46 | @echo " linkcheck to check all external links for integrity" 47 | @echo " doctest to run all doctests embedded in the documentation (if enabled)" 48 | 49 | clean: 50 | rm -rf $(BUILDDIR)/* 51 | 52 | html: 53 | $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html 54 | @echo 55 | @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." 56 | 57 | dirhtml: 58 | $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml 59 | @echo 60 | @echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml." 61 | 62 | singlehtml: 63 | $(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml 64 | @echo 65 | @echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml." 66 | 67 | pickle: 68 | $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle 69 | @echo 70 | @echo "Build finished; now you can process the pickle files." 71 | 72 | json: 73 | $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json 74 | @echo 75 | @echo "Build finished; now you can process the JSON files." 76 | 77 | htmlhelp: 78 | $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp 79 | @echo 80 | @echo "Build finished; now you can run HTML Help Workshop with the" \ 81 | ".hhp project file in $(BUILDDIR)/htmlhelp." 82 | 83 | qthelp: 84 | $(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp 85 | @echo 86 | @echo "Build finished; now you can run "qcollectiongenerator" with the" \ 87 | ".qhcp project file in $(BUILDDIR)/qthelp, like this:" 88 | @echo "# qcollectiongenerator $(BUILDDIR)/qthelp/JSGFGrammarTools.qhcp" 89 | @echo "To view the help file:" 90 | @echo "# assistant -collectionFile $(BUILDDIR)/qthelp/JSGFGrammarTools.qhc" 91 | 92 | devhelp: 93 | $(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp 94 | @echo 95 | @echo "Build finished." 96 | @echo "To view the help file:" 97 | @echo "# mkdir -p $$HOME/.local/share/devhelp/JSGFGrammarTools" 98 | @echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/JSGFGrammarTools" 99 | @echo "# devhelp" 100 | 101 | epub: 102 | $(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub 103 | @echo 104 | @echo "Build finished. The epub file is in $(BUILDDIR)/epub." 105 | 106 | latex: 107 | $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex 108 | @echo 109 | @echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex." 110 | @echo "Run \`make' in that directory to run these through (pdf)latex" \ 111 | "(use \`make latexpdf' here to do that automatically)." 112 | 113 | latexpdf: 114 | $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex 115 | @echo "Running LaTeX files through pdflatex..." 116 | $(MAKE) -C $(BUILDDIR)/latex all-pdf 117 | @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." 118 | 119 | latexpdfja: 120 | $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex 121 | @echo "Running LaTeX files through platex and dvipdfmx..." 122 | $(MAKE) -C $(BUILDDIR)/latex all-pdf-ja 123 | @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." 124 | 125 | text: 126 | $(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text 127 | @echo 128 | @echo "Build finished. The text files are in $(BUILDDIR)/text." 129 | 130 | man: 131 | $(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man 132 | @echo 133 | @echo "Build finished. The manual pages are in $(BUILDDIR)/man." 134 | 135 | texinfo: 136 | $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo 137 | @echo 138 | @echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo." 139 | @echo "Run \`make' in that directory to run these through makeinfo" \ 140 | "(use \`make info' here to do that automatically)." 141 | 142 | info: 143 | $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo 144 | @echo "Running Texinfo files through makeinfo..." 145 | make -C $(BUILDDIR)/texinfo info 146 | @echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo." 147 | 148 | gettext: 149 | $(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale 150 | @echo 151 | @echo "Build finished. The message catalogs are in $(BUILDDIR)/locale." 152 | 153 | changes: 154 | $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes 155 | @echo 156 | @echo "The overview file is in $(BUILDDIR)/changes." 157 | 158 | linkcheck: 159 | $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck 160 | @echo 161 | @echo "Link check complete; look for any errors in the above output " \ 162 | "or in $(BUILDDIR)/linkcheck/output.txt." 163 | 164 | doctest: 165 | $(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest 166 | @echo "Testing of doctests in the sources finished, look at the " \ 167 | "results in $(BUILDDIR)/doctest/output.txt." 168 | 169 | xml: 170 | $(SPHINXBUILD) -b xml $(ALLSPHINXOPTS) $(BUILDDIR)/xml 171 | @echo 172 | @echo "Build finished. The XML files are in $(BUILDDIR)/xml." 173 | 174 | pseudoxml: 175 | $(SPHINXBUILD) -b pseudoxml $(ALLSPHINXOPTS) $(BUILDDIR)/pseudoxml 176 | @echo 177 | @echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml." 178 | -------------------------------------------------------------------------------- /docs/_build/html/_static/doctools.js: -------------------------------------------------------------------------------- 1 | /* 2 | * doctools.js 3 | * ~~~~~~~~~~~ 4 | * 5 | * Sphinx JavaScript utilities for all documentation. 6 | * 7 | * :copyright: Copyright 2007-2014 by the Sphinx team, see AUTHORS. 8 | * :license: BSD, see LICENSE for details. 9 | * 10 | */ 11 | 12 | /** 13 | * select a different prefix for underscore 14 | */ 15 | $u = _.noConflict(); 16 | 17 | /** 18 | * make the code below compatible with browsers without 19 | * an installed firebug like debugger 20 | if (!window.console || !console.firebug) { 21 | var names = ["log", "debug", "info", "warn", "error", "assert", "dir", 22 | "dirxml", "group", "groupEnd", "time", "timeEnd", "count", "trace", 23 | "profile", "profileEnd"]; 24 | window.console = {}; 25 | for (var i = 0; i < names.length; ++i) 26 | window.console[names[i]] = function() {}; 27 | } 28 | */ 29 | 30 | /** 31 | * small helper function to urldecode strings 32 | */ 33 | jQuery.urldecode = function(x) { 34 | return decodeURIComponent(x).replace(/\+/g, ' '); 35 | }; 36 | 37 | /** 38 | * small helper function to urlencode strings 39 | */ 40 | jQuery.urlencode = encodeURIComponent; 41 | 42 | /** 43 | * This function returns the parsed url parameters of the 44 | * current request. Multiple values per key are supported, 45 | * it will always return arrays of strings for the value parts. 46 | */ 47 | jQuery.getQueryParameters = function(s) { 48 | if (typeof s == 'undefined') 49 | s = document.location.search; 50 | var parts = s.substr(s.indexOf('?') + 1).split('&'); 51 | var result = {}; 52 | for (var i = 0; i < parts.length; i++) { 53 | var tmp = parts[i].split('=', 2); 54 | var key = jQuery.urldecode(tmp[0]); 55 | var value = jQuery.urldecode(tmp[1]); 56 | if (key in result) 57 | result[key].push(value); 58 | else 59 | result[key] = [value]; 60 | } 61 | return result; 62 | }; 63 | 64 | /** 65 | * highlight a given string on a jquery object by wrapping it in 66 | * span elements with the given class name. 67 | */ 68 | jQuery.fn.highlightText = function(text, className) { 69 | function highlight(node) { 70 | if (node.nodeType == 3) { 71 | var val = node.nodeValue; 72 | var pos = val.toLowerCase().indexOf(text); 73 | if (pos >= 0 && !jQuery(node.parentNode).hasClass(className)) { 74 | var span = document.createElement("span"); 75 | span.className = className; 76 | span.appendChild(document.createTextNode(val.substr(pos, text.length))); 77 | node.parentNode.insertBefore(span, node.parentNode.insertBefore( 78 | document.createTextNode(val.substr(pos + text.length)), 79 | node.nextSibling)); 80 | node.nodeValue = val.substr(0, pos); 81 | } 82 | } 83 | else if (!jQuery(node).is("button, select, textarea")) { 84 | jQuery.each(node.childNodes, function() { 85 | highlight(this); 86 | }); 87 | } 88 | } 89 | return this.each(function() { 90 | highlight(this); 91 | }); 92 | }; 93 | 94 | /** 95 | * Small JavaScript module for the documentation. 96 | */ 97 | var Documentation = { 98 | 99 | init : function() { 100 | this.fixFirefoxAnchorBug(); 101 | this.highlightSearchWords(); 102 | this.initIndexTable(); 103 | }, 104 | 105 | /** 106 | * i18n support 107 | */ 108 | TRANSLATIONS : {}, 109 | PLURAL_EXPR : function(n) { return n == 1 ? 0 : 1; }, 110 | LOCALE : 'unknown', 111 | 112 | // gettext and ngettext don't access this so that the functions 113 | // can safely bound to a different name (_ = Documentation.gettext) 114 | gettext : function(string) { 115 | var translated = Documentation.TRANSLATIONS[string]; 116 | if (typeof translated == 'undefined') 117 | return string; 118 | return (typeof translated == 'string') ? translated : translated[0]; 119 | }, 120 | 121 | ngettext : function(singular, plural, n) { 122 | var translated = Documentation.TRANSLATIONS[singular]; 123 | if (typeof translated == 'undefined') 124 | return (n == 1) ? singular : plural; 125 | return translated[Documentation.PLURALEXPR(n)]; 126 | }, 127 | 128 | addTranslations : function(catalog) { 129 | for (var key in catalog.messages) 130 | this.TRANSLATIONS[key] = catalog.messages[key]; 131 | this.PLURAL_EXPR = new Function('n', 'return +(' + catalog.plural_expr + ')'); 132 | this.LOCALE = catalog.locale; 133 | }, 134 | 135 | /** 136 | * add context elements like header anchor links 137 | */ 138 | addContextElements : function() { 139 | $('div[id] > :header:first').each(function() { 140 | $('\u00B6'). 141 | attr('href', '#' + this.id). 142 | attr('title', _('Permalink to this headline')). 143 | appendTo(this); 144 | }); 145 | $('dt[id]').each(function() { 146 | $('\u00B6'). 147 | attr('href', '#' + this.id). 148 | attr('title', _('Permalink to this definition')). 149 | appendTo(this); 150 | }); 151 | }, 152 | 153 | /** 154 | * workaround a firefox stupidity 155 | */ 156 | fixFirefoxAnchorBug : function() { 157 | if (document.location.hash && $.browser.mozilla) 158 | window.setTimeout(function() { 159 | document.location.href += ''; 160 | }, 10); 161 | }, 162 | 163 | /** 164 | * highlight the search words provided in the url in the text 165 | */ 166 | highlightSearchWords : function() { 167 | var params = $.getQueryParameters(); 168 | var terms = (params.highlight) ? params.highlight[0].split(/\s+/) : []; 169 | if (terms.length) { 170 | var body = $('div.body'); 171 | if (!body.length) { 172 | body = $('body'); 173 | } 174 | window.setTimeout(function() { 175 | $.each(terms, function() { 176 | body.highlightText(this.toLowerCase(), 'highlighted'); 177 | }); 178 | }, 10); 179 | $('

' + _('Hide Search Matches') + '

') 181 | .appendTo($('#searchbox')); 182 | } 183 | }, 184 | 185 | /** 186 | * init the domain index toggle buttons 187 | */ 188 | initIndexTable : function() { 189 | var togglers = $('img.toggler').click(function() { 190 | var src = $(this).attr('src'); 191 | var idnum = $(this).attr('id').substr(7); 192 | $('tr.cg-' + idnum).toggle(); 193 | if (src.substr(-9) == 'minus.png') 194 | $(this).attr('src', src.substr(0, src.length-9) + 'plus.png'); 195 | else 196 | $(this).attr('src', src.substr(0, src.length-8) + 'minus.png'); 197 | }).css('display', ''); 198 | if (DOCUMENTATION_OPTIONS.COLLAPSE_INDEX) { 199 | togglers.click(); 200 | } 201 | }, 202 | 203 | /** 204 | * helper function to hide the search marks again 205 | */ 206 | hideSearchWords : function() { 207 | $('#searchbox .highlight-link').fadeOut(300); 208 | $('span.highlighted').removeClass('highlighted'); 209 | }, 210 | 211 | /** 212 | * make the url absolute 213 | */ 214 | makeURL : function(relativeURL) { 215 | return DOCUMENTATION_OPTIONS.URL_ROOT + '/' + relativeURL; 216 | }, 217 | 218 | /** 219 | * get the current relative url 220 | */ 221 | getCurrentURL : function() { 222 | var path = document.location.pathname; 223 | var parts = path.split(/\//); 224 | $.each(DOCUMENTATION_OPTIONS.URL_ROOT.split(/\//), function() { 225 | if (this == '..') 226 | parts.pop(); 227 | }); 228 | var url = parts.join('/'); 229 | return path.substring(url.lastIndexOf('/') + 1, path.length - 1); 230 | } 231 | }; 232 | 233 | // quick alias for translations 234 | _ = Documentation.gettext; 235 | 236 | $(document).ready(function() { 237 | Documentation.init(); 238 | }); 239 | -------------------------------------------------------------------------------- /jsgf/cli.py: -------------------------------------------------------------------------------- 1 | """ 2 | Command-line interface for JSGF Tools. 3 | 4 | This module provides clean CLI commands that use the modern JSGF API. 5 | """ 6 | 7 | import argparse 8 | import sys 9 | from pathlib import Path 10 | from typing import Optional 11 | 12 | from .grammar import Grammar 13 | from .generators import DeterministicGenerator, ProbabilisticGenerator, GeneratorConfig 14 | from .exceptions import JSGFError 15 | 16 | 17 | def deterministic_command(): 18 | """Command-line interface for deterministic generation.""" 19 | parser = argparse.ArgumentParser( 20 | description='Generate all possible strings from a non-recursive JSGF grammar', 21 | formatter_class=argparse.RawDescriptionHelpFormatter, 22 | epilog=''' 23 | Examples: 24 | %(prog)s grammar.jsgf 25 | %(prog)s grammar.jsgf --rule greeting 26 | %(prog)s grammar.jsgf --max-results 100 27 | ''' 28 | ) 29 | 30 | parser.add_argument( 31 | 'grammar_file', 32 | help='Path to the JSGF grammar file' 33 | ) 34 | 35 | parser.add_argument( 36 | '--rule', '-r', 37 | help='Specific rule to generate from (default: all public rules)' 38 | ) 39 | 40 | parser.add_argument( 41 | '--max-results', '-m', 42 | type=int, 43 | help='Maximum number of strings to generate' 44 | ) 45 | 46 | parser.add_argument( 47 | '--max-recursion', '-d', 48 | type=int, 49 | default=50, 50 | help='Maximum recursion depth (default: 50)' 51 | ) 52 | 53 | parser.add_argument( 54 | '--output', '-o', 55 | help='Output file (default: stdout)' 56 | ) 57 | 58 | args = parser.parse_args() 59 | 60 | try: 61 | # Load grammar 62 | grammar = Grammar.from_file(args.grammar_file) 63 | 64 | # Check for recursion if no specific rule is given 65 | if not args.rule and grammar.is_recursive(): 66 | print( 67 | "Warning: Grammar contains recursive rules. " 68 | "Consider using probabilistic generation instead.", 69 | file=sys.stderr 70 | ) 71 | 72 | # Create generator 73 | config = GeneratorConfig( 74 | max_recursion_depth=args.max_recursion, 75 | max_results=args.max_results 76 | ) 77 | generator = DeterministicGenerator(grammar, config) 78 | 79 | # Open output file if specified 80 | output_file = open(args.output, 'w') if args.output else sys.stdout 81 | 82 | try: 83 | # Generate strings 84 | for string in generator.generate(args.rule): 85 | print(string, file=output_file) 86 | finally: 87 | if args.output: 88 | output_file.close() 89 | 90 | except JSGFError as e: 91 | print(f"Error: {e}", file=sys.stderr) 92 | sys.exit(1) 93 | except FileNotFoundError as e: 94 | print(f"Error: {e}", file=sys.stderr) 95 | sys.exit(1) 96 | except KeyboardInterrupt: 97 | print("\\nGeneration interrupted", file=sys.stderr) 98 | sys.exit(1) 99 | except Exception as e: 100 | print(f"Unexpected error: {e}", file=sys.stderr) 101 | sys.exit(1) 102 | 103 | 104 | def probabilistic_command(): 105 | """Command-line interface for probabilistic generation.""" 106 | parser = argparse.ArgumentParser( 107 | description='Generate random strings from a JSGF grammar', 108 | formatter_class=argparse.RawDescriptionHelpFormatter, 109 | epilog=''' 110 | Examples: 111 | %(prog)s grammar.jsgf 10 112 | %(prog)s grammar.jsgf 20 --rule greeting 113 | %(prog)s grammar.jsgf 5 --seed 42 114 | ''' 115 | ) 116 | 117 | parser.add_argument( 118 | 'grammar_file', 119 | help='Path to the JSGF grammar file' 120 | ) 121 | 122 | parser.add_argument( 123 | 'count', 124 | type=int, 125 | help='Number of strings to generate' 126 | ) 127 | 128 | parser.add_argument( 129 | '--rule', '-r', 130 | help='Specific rule to generate from (default: all public rules)' 131 | ) 132 | 133 | parser.add_argument( 134 | '--seed', '-s', 135 | type=int, 136 | help='Random seed for reproducible results' 137 | ) 138 | 139 | parser.add_argument( 140 | '--max-recursion', '-d', 141 | type=int, 142 | default=50, 143 | help='Maximum recursion depth (default: 50)' 144 | ) 145 | 146 | parser.add_argument( 147 | '--output', '-o', 148 | help='Output file (default: stdout)' 149 | ) 150 | 151 | args = parser.parse_args() 152 | 153 | try: 154 | # Load grammar 155 | grammar = Grammar.from_file(args.grammar_file) 156 | 157 | # Create generator 158 | config = GeneratorConfig( 159 | max_recursion_depth=args.max_recursion, 160 | random_seed=args.seed 161 | ) 162 | generator = ProbabilisticGenerator(grammar, config) 163 | 164 | # Open output file if specified 165 | output_file = open(args.output, 'w') if args.output else sys.stdout 166 | 167 | try: 168 | # Generate specified number of strings 169 | strings = generator.generate_list(args.rule, args.count) 170 | for string in strings: 171 | print(string, file=output_file) 172 | finally: 173 | if args.output: 174 | output_file.close() 175 | 176 | except JSGFError as e: 177 | print(f"Error: {e}", file=sys.stderr) 178 | sys.exit(1) 179 | except FileNotFoundError as e: 180 | print(f"Error: {e}", file=sys.stderr) 181 | sys.exit(1) 182 | except KeyboardInterrupt: 183 | print("\\nGeneration interrupted", file=sys.stderr) 184 | sys.exit(1) 185 | except Exception as e: 186 | print(f"Unexpected error: {e}", file=sys.stderr) 187 | sys.exit(1) 188 | 189 | 190 | def grammar_info_command(): 191 | """Command-line interface for grammar information.""" 192 | parser = argparse.ArgumentParser( 193 | description='Display information about a JSGF grammar', 194 | formatter_class=argparse.RawDescriptionHelpFormatter 195 | ) 196 | 197 | parser.add_argument( 198 | 'grammar_file', 199 | help='Path to the JSGF grammar file' 200 | ) 201 | 202 | parser.add_argument( 203 | '--verbose', '-v', 204 | action='store_true', 205 | help='Show detailed information' 206 | ) 207 | 208 | args = parser.parse_args() 209 | 210 | try: 211 | # Load grammar 212 | grammar = Grammar.from_file(args.grammar_file) 213 | 214 | # Basic information 215 | print(f"Grammar: {args.grammar_file}") 216 | print(f"Total rules: {len(grammar)}") 217 | print(f"Public rules: {len(grammar.public_rules)}") 218 | 219 | if args.verbose: 220 | print("\\nPublic rules:") 221 | for rule in grammar.public_rules: 222 | print(f" - {rule.name}") 223 | 224 | print("\\nAll rules:") 225 | for rule_name in sorted(grammar.rule_names): 226 | rule = grammar.get_rule(rule_name) 227 | visibility = "public" if rule.is_public else "private" 228 | print(f" - {rule_name} ({visibility})") 229 | 230 | # Check for recursion 231 | if grammar.is_recursive(): 232 | cycles = grammar.detect_cycles() 233 | print(f"\\nRecursive: Yes ({len(cycles)} cycle(s))") 234 | if args.verbose: 235 | for i, cycle in enumerate(cycles, 1): 236 | print(f" Cycle {i}: {' -> '.join(cycle)}") 237 | else: 238 | print("\\nRecursive: No") 239 | 240 | # Validation 241 | try: 242 | grammar.validate() 243 | print("Validation: Passed") 244 | except Exception as e: 245 | print(f"Validation: Failed - {e}") 246 | 247 | except JSGFError as e: 248 | print(f"Error: {e}", file=sys.stderr) 249 | sys.exit(1) 250 | except FileNotFoundError as e: 251 | print(f"Error: {e}", file=sys.stderr) 252 | sys.exit(1) 253 | except Exception as e: 254 | print(f"Unexpected error: {e}", file=sys.stderr) 255 | sys.exit(1) -------------------------------------------------------------------------------- /docs/_build/html/JSGFGrammar.html: -------------------------------------------------------------------------------- 1 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | JSGFGrammar module — JSGF Grammar Tools 1.0 documentation 10 | 11 | 12 | 13 | 14 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 |
32 |

Navigation

33 | 48 |
49 | 50 |
51 |
52 |
53 |
54 | 55 |
56 |

JSGFGrammar module

57 |

This file lays out the class structure for a JSGF Grammar.

58 |
59 |
60 | class JSGFGrammar.Disjunction(disjuncts)
61 |

Disjunction class stores disjuncts in a list

62 |
63 | 64 |
65 |
66 | class JSGFGrammar.Grammar
67 |

Grammar class which contains a list for public rules and a list 68 | for all rules.

69 |
70 |
71 | addPublicRule(rule)
72 |

adds a rule to the list of public rules

73 |
74 | 75 |
76 |
77 | addRule(rule)
78 |

adds a rule to the list of rules

79 |
80 | 81 |
82 |
83 | getRHS(nt)
84 |

returns rule definition

85 | 86 | 87 | 88 | 89 | 90 | 91 | 92 |
Parameters:nt – Non-Terminal (variable) whose definition to get
93 |
94 | 95 |
96 | 97 |
98 |
99 | class JSGFGrammar.JSGFExpression
100 |
101 | 102 |
103 |
104 | class JSGFGrammar.NonTerminal(ntName)
105 |

NonTerminal class simply stores the label of the nonterminal

106 |
107 | 108 |
109 |
110 | class JSGFGrammar.Optional(option)
111 |

Optional class stores either a JSGFExpression, list, or string 112 | as its optional element

113 |
114 | 115 |
116 |
117 | class JSGFGrammar.Rule(lhs, rhs)
118 |

Rule class, represents a JSGF rule, with a nonterminal name representing the 119 | left hand side, and a list of possible expansions representing the right 120 | hand side.

121 |
122 | 123 |
124 | 125 | 126 |
127 |
128 |
129 |
130 |
131 |

Previous topic

132 |

Welcome to JSGF Grammar Tools’s documentation!

134 |

Next topic

135 |

JSGFParser module

137 |

This Page

138 | 142 | 154 | 155 |
156 |
157 |
158 |
159 |
160 |

Navigation

161 | 176 |
177 |
178 | © Copyright 2014, Timothy Ho, Nicholas Bacuez. 179 | Created using Sphinx 1.2.2. 180 |
181 | 182 | -------------------------------------------------------------------------------- /docs/conf.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # 3 | # JSGF Grammar Tools documentation build configuration file, created by 4 | # sphinx-quickstart on Thu May 29 15:47:59 2014. 5 | # 6 | # This file is execfile()d with the current directory set to its 7 | # containing dir. 8 | # 9 | # Note that not all possible configuration values are present in this 10 | # autogenerated file. 11 | # 12 | # All configuration values have a default; values that are commented out 13 | # serve to show the default. 14 | 15 | import sys 16 | import os 17 | 18 | # If extensions (or modules to document with autodoc) are in another directory, 19 | # add these directories to sys.path here. If the directory is relative to the 20 | # documentation root, use os.path.abspath to make it absolute, like shown here. 21 | sys.path.insert(0, os.path.abspath('../')) 22 | 23 | # -- General configuration ------------------------------------------------ 24 | 25 | # If your documentation needs a minimal Sphinx version, state it here. 26 | #needs_sphinx = '1.0' 27 | 28 | # Add any Sphinx extension module names here, as strings. They can be 29 | # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom 30 | # ones. 31 | extensions = [ 32 | 'sphinx.ext.autodoc', 33 | ] 34 | 35 | # Add any paths that contain templates here, relative to this directory. 36 | templates_path = ['_templates'] 37 | 38 | # The suffix of source filenames. 39 | source_suffix = '.rst' 40 | 41 | # The encoding of source files. 42 | #source_encoding = 'utf-8-sig' 43 | 44 | # The master toctree document. 45 | master_doc = 'index' 46 | 47 | # General information about the project. 48 | project = u'JSGF Grammar Tools' 49 | copyright = u'2014, Timothy Ho, Nicholas Bacuez' 50 | 51 | # The version info for the project you're documenting, acts as replacement for 52 | # |version| and |release|, also used in various other places throughout the 53 | # built documents. 54 | # 55 | # The short X.Y version. 56 | version = '1.0' 57 | # The full version, including alpha/beta/rc tags. 58 | release = '1.0' 59 | 60 | # The language for content autogenerated by Sphinx. Refer to documentation 61 | # for a list of supported languages. 62 | #language = None 63 | 64 | # There are two options for replacing |today|: either, you set today to some 65 | # non-false value, then it is used: 66 | #today = '' 67 | # Else, today_fmt is used as the format for a strftime call. 68 | #today_fmt = '%B %d, %Y' 69 | 70 | # List of patterns, relative to source directory, that match files and 71 | # directories to ignore when looking for source files. 72 | exclude_patterns = ['_build'] 73 | 74 | # The reST default role (used for this markup: `text`) to use for all 75 | # documents. 76 | #default_role = None 77 | 78 | # If true, '()' will be appended to :func: etc. cross-reference text. 79 | #add_function_parentheses = True 80 | 81 | # If true, the current module name will be prepended to all description 82 | # unit titles (such as .. function::). 83 | #add_module_names = True 84 | 85 | # If true, sectionauthor and moduleauthor directives will be shown in the 86 | # output. They are ignored by default. 87 | #show_authors = False 88 | 89 | # The name of the Pygments (syntax highlighting) style to use. 90 | pygments_style = 'sphinx' 91 | 92 | # A list of ignored prefixes for module index sorting. 93 | #modindex_common_prefix = [] 94 | 95 | # If true, keep warnings as "system message" paragraphs in the built documents. 96 | #keep_warnings = False 97 | 98 | 99 | # -- Options for HTML output ---------------------------------------------- 100 | 101 | # The theme to use for HTML and HTML Help pages. See the documentation for 102 | # a list of builtin themes. 103 | html_theme = 'default' 104 | 105 | # Theme options are theme-specific and customize the look and feel of a theme 106 | # further. For a list of options available for each theme, see the 107 | # documentation. 108 | #html_theme_options = {} 109 | 110 | # Add any paths that contain custom themes here, relative to this directory. 111 | #html_theme_path = [] 112 | 113 | # The name for this set of Sphinx documents. If None, it defaults to 114 | # " v documentation". 115 | #html_title = None 116 | 117 | # A shorter title for the navigation bar. Default is the same as html_title. 118 | #html_short_title = None 119 | 120 | # The name of an image file (relative to this directory) to place at the top 121 | # of the sidebar. 122 | #html_logo = None 123 | 124 | # The name of an image file (within the static path) to use as favicon of the 125 | # docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 126 | # pixels large. 127 | #html_favicon = None 128 | 129 | # Add any paths that contain custom static files (such as style sheets) here, 130 | # relative to this directory. They are copied after the builtin static files, 131 | # so a file named "default.css" will overwrite the builtin "default.css". 132 | html_static_path = ['_static'] 133 | 134 | # Add any extra paths that contain custom files (such as robots.txt or 135 | # .htaccess) here, relative to this directory. These files are copied 136 | # directly to the root of the documentation. 137 | #html_extra_path = [] 138 | 139 | # If not '', a 'Last updated on:' timestamp is inserted at every page bottom, 140 | # using the given strftime format. 141 | #html_last_updated_fmt = '%b %d, %Y' 142 | 143 | # If true, SmartyPants will be used to convert quotes and dashes to 144 | # typographically correct entities. 145 | #html_use_smartypants = True 146 | 147 | # Custom sidebar templates, maps document names to template names. 148 | #html_sidebars = {} 149 | 150 | # Additional templates that should be rendered to pages, maps page names to 151 | # template names. 152 | #html_additional_pages = {} 153 | 154 | # If false, no module index is generated. 155 | #html_domain_indices = True 156 | 157 | # If false, no index is generated. 158 | #html_use_index = True 159 | 160 | # If true, the index is split into individual pages for each letter. 161 | #html_split_index = False 162 | 163 | # If true, links to the reST sources are added to the pages. 164 | #html_show_sourcelink = True 165 | 166 | # If true, "Created using Sphinx" is shown in the HTML footer. Default is True. 167 | #html_show_sphinx = True 168 | 169 | # If true, "(C) Copyright ..." is shown in the HTML footer. Default is True. 170 | #html_show_copyright = True 171 | 172 | # If true, an OpenSearch description file will be output, and all pages will 173 | # contain a tag referring to it. The value of this option must be the 174 | # base URL from which the finished HTML is served. 175 | #html_use_opensearch = '' 176 | 177 | # This is the file name suffix for HTML files (e.g. ".xhtml"). 178 | #html_file_suffix = None 179 | 180 | # Output file base name for HTML help builder. 181 | htmlhelp_basename = 'JSGFGrammarToolsdoc' 182 | 183 | 184 | # -- Options for LaTeX output --------------------------------------------- 185 | 186 | latex_elements = { 187 | # The paper size ('letterpaper' or 'a4paper'). 188 | #'papersize': 'letterpaper', 189 | 190 | # The font size ('10pt', '11pt' or '12pt'). 191 | #'pointsize': '10pt', 192 | 193 | # Additional stuff for the LaTeX preamble. 194 | #'preamble': '', 195 | } 196 | 197 | # Grouping the document tree into LaTeX files. List of tuples 198 | # (source start file, target name, title, 199 | # author, documentclass [howto, manual, or own class]). 200 | latex_documents = [ 201 | ('index', 'JSGFGrammarTools.tex', u'JSGF Grammar Tools Documentation', 202 | u'Timothy Ho, Nicholas Bacuez', 'manual'), 203 | ] 204 | 205 | # The name of an image file (relative to this directory) to place at the top of 206 | # the title page. 207 | #latex_logo = None 208 | 209 | # For "manual" documents, if this is true, then toplevel headings are parts, 210 | # not chapters. 211 | #latex_use_parts = False 212 | 213 | # If true, show page references after internal links. 214 | #latex_show_pagerefs = False 215 | 216 | # If true, show URL addresses after external links. 217 | #latex_show_urls = False 218 | 219 | # Documents to append as an appendix to all manuals. 220 | #latex_appendices = [] 221 | 222 | # If false, no module index is generated. 223 | #latex_domain_indices = True 224 | 225 | 226 | # -- Options for manual page output --------------------------------------- 227 | 228 | # One entry per manual page. List of tuples 229 | # (source start file, name, description, authors, manual section). 230 | man_pages = [ 231 | ('index', 'jsgfgrammartools', u'JSGF Grammar Tools Documentation', 232 | [u'Timothy Ho, Nicholas Bacuez'], 1) 233 | ] 234 | 235 | # If true, show URL addresses after external links. 236 | #man_show_urls = False 237 | 238 | 239 | # -- Options for Texinfo output ------------------------------------------- 240 | 241 | # Grouping the document tree into Texinfo files. List of tuples 242 | # (source start file, target name, title, author, 243 | # dir menu entry, description, category) 244 | texinfo_documents = [ 245 | ('index', 'JSGFGrammarTools', u'JSGF Grammar Tools Documentation', 246 | u'Timothy Ho, Nicholas Bacuez', 'JSGFGrammarTools', 'One line description of project.', 247 | 'Miscellaneous'), 248 | ] 249 | 250 | # Documents to append as an appendix to all manuals. 251 | #texinfo_appendices = [] 252 | 253 | # If false, no module index is generated. 254 | #texinfo_domain_indices = True 255 | 256 | # How to display URL addresses: 'footnote', 'no', or 'inline'. 257 | #texinfo_show_urls = 'footnote' 258 | 259 | # If true, do not generate a @detailmenu in the "Top" node's menu. 260 | #texinfo_no_detailmenu = False 261 | -------------------------------------------------------------------------------- /docs/_build/html/DeterministicGenerator.html: -------------------------------------------------------------------------------- 1 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | Deterministic Generator module — JSGF Grammar Tools 1.0 documentation 10 | 11 | 12 | 13 | 14 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 |
31 |

Navigation

32 | 44 |
45 | 46 |
47 |
48 |
49 |
50 | 51 |
52 |

Deterministic Generator module

53 |

This file deterministically generates strings from a JSGF Grammar, whether there are weights defined in rules or not. It requires one argument: the path to the JSGF Grammar file. You can run this on the included grammar IdeasNonRecursive.gram:

54 |
55 |
python DeterministicGenerator.py IdeasNonRecursive.gram
56 |

This will generate all strings defined by the public rules of IdeasNonRecursive.gram. It is important that the grammar used by the generator is not recursive (rules should not directly or indirectly reference themselves), so that the generator terminates. Otherwise, you may get a maximum recursion depth exceeded error or a segmentation fault.

57 |
58 |
59 | DeterministicGenerator.combineSets(listOfSets)
60 |

Combines sets of strings by taking the cross product of the sets and concatenating the elements in the resulting tuples

61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 |
Parameters:listOfSets – 2-D list of strings
Returns:a list of strings
71 |
72 | 73 |
74 |
75 | DeterministicGenerator.processDisjunction(disj)
76 |

Returns the string representations of a set of alternatives

77 | 78 | 79 | 80 | 81 | 82 | 83 | 84 |
Returns:list of strings, where each string is each alternative
85 |
86 | 87 |
88 |
89 | DeterministicGenerator.processNonTerminal(nt)
90 |

Finds the rule expansion for a nonterminal and returns its expansion.

91 |
92 | 93 |
94 |
95 | DeterministicGenerator.processOptional(opt)
96 |

Returns the string representations of an optional grouping

97 | 98 | 99 | 100 | 101 | 102 | 103 | 104 |
Returns:list of strings, including an empty string
105 |
106 | 107 |
108 |
109 | DeterministicGenerator.processRHS(rhs)
110 |

Depending on the type of the argument, calls the corresponding 111 | function to deal with that type.

112 | 113 | 114 | 115 | 116 | 117 | 118 | 119 | 120 | 121 |
Parameters:rhs (either a JSGF Expression, list, or string) – portion of JSGF rule
Returns:list of strings
122 |
123 | 124 |
125 |
126 | DeterministicGenerator.processSequence(seq)
127 |

Combines adjacent elements in a sequence

128 |
129 | 130 |
131 | 132 | 133 |
134 |
135 |
136 |
137 |
138 |

Previous topic

139 |

Probabilistic Generator module

141 |

This Page

142 | 146 | 158 | 159 |
160 |
161 |
162 |
163 |
164 |

Navigation

165 | 177 |
178 |
179 | © Copyright 2014, Timothy Ho, Nicholas Bacuez. 180 | Created using Sphinx 1.2.2. 181 |
182 | 183 | -------------------------------------------------------------------------------- /docs/_build/html/ProbabilisticGenerator.html: -------------------------------------------------------------------------------- 1 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | Probabilistic Generator module — JSGF Grammar Tools 1.0 documentation 10 | 11 | 12 | 13 | 14 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 |
32 |

Navigation

33 | 48 |
49 | 50 |
51 |
52 |
53 |
54 | 55 |
56 |

Probabilistic Generator module

57 |

This file probabilistically generates strings from a JSGF grammar. It takes advantage of weights assigned to alternatives (separated by pipes) by choosing to expand higher weighted alternatives with greater probability. For sets of alternatives without weights, each alternative is equally likely to be expanded. For optional groups, the elements in the group have a 50% chance of being expanded.

58 |

It requires two arguments: the path to the JSGF Grammar file, and the number of strings to generate. You can run this on the included grammar Ideas.gram:

59 |
60 |
python ProbabilisticGenerator.py Ideas.gram 20
61 |

This will generate 20 sentences based on the public rule(s) in Ideas.gram, using the weights if they are provided.

62 |
63 |
64 | ProbabilisticGenerator.combineSets(listOfSets)
65 |

Combines sets of strings by taking the cross product of the sets and concatenating the elements in the resulting tuples

66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 |
Parameters:listOfSets – 2-D list of strings
Returns:a list of strings
76 |
77 | 78 |
79 |
80 | ProbabilisticGenerator.processDisjunction(disj)
81 |

Chooses either a random disjunct (for alternatives without weights) or 82 | a disjunct based on defined weights.

83 |
84 | 85 |
86 |
87 | ProbabilisticGenerator.processNonTerminal(nt)
88 |

Finds the rule expansion for a nonterminal and returns its expansion.

89 |
90 | 91 |
92 |
93 | ProbabilisticGenerator.processOptional(opt)
94 |

Processes the optional element 50% of the time, skips it the other 50% of the time

95 |
96 | 97 |
98 |
99 | ProbabilisticGenerator.processRHS(rhs)
100 |
101 | 102 |
103 |
104 | ProbabilisticGenerator.processSequence(seq)
105 |

Combines adjacent elements in a sequence.

106 |
107 | 108 |
109 |
110 | ProbabilisticGenerator.weightedChoice(listOfTuples)
111 |

Chooses an element of a list based on its weight

112 | 113 | 114 | 115 | 116 | 117 | 118 | 119 | 120 | 121 |
Parameters:listOfTuples – a list of (element, weight) tuples, where the element can be a JSGF expression object, string, or list, and the weight is a float
Returns:the first element of a chosen tuple
122 |
123 | 124 |
125 | 126 | 127 |
128 |
129 |
130 |
131 |
132 |

Previous topic

133 |

JSGFParser module

135 |

Next topic

136 |

Deterministic Generator module

138 |

This Page

139 | 143 | 155 | 156 |
157 |
158 |
159 |
160 |
161 |

Navigation

162 | 177 |
178 |
179 | © Copyright 2014, Timothy Ho, Nicholas Bacuez. 180 | Created using Sphinx 1.2.2. 181 |
182 | 183 | -------------------------------------------------------------------------------- /jsgf/grammar.py: -------------------------------------------------------------------------------- 1 | """ 2 | JSGF Grammar representation and parsing functionality. 3 | """ 4 | 5 | from typing import Dict, List, Optional, Union, TextIO, Iterator 6 | from pathlib import Path 7 | import re 8 | from io import StringIO 9 | 10 | from .ast_nodes import ( 11 | JSGFNode, Terminal, NonTerminal, Sequence, Alternative, 12 | Optional as OptionalNode, Group, Rule 13 | ) 14 | from .exceptions import ParseError, ValidationError 15 | from .legacy_adapter import LegacyAdapter 16 | 17 | 18 | class Grammar: 19 | """ 20 | Represents a complete JSGF grammar with rules and provides parsing functionality. 21 | 22 | This class encapsulates all grammar rules and provides methods for parsing, 23 | validation, and rule lookup. 24 | """ 25 | 26 | def __init__(self): 27 | self._rules: Dict[str, Rule] = {} 28 | self._public_rules: List[Rule] = [] 29 | 30 | @classmethod 31 | def from_string(cls, grammar_text: str) -> 'Grammar': 32 | """ 33 | Parse a grammar from a string. 34 | 35 | Args: 36 | grammar_text: The JSGF grammar text to parse 37 | 38 | Returns: 39 | A Grammar instance 40 | 41 | Raises: 42 | ParseError: If the grammar cannot be parsed 43 | """ 44 | grammar = cls() 45 | adapter = LegacyAdapter() 46 | 47 | try: 48 | with StringIO(grammar_text) as f: 49 | adapter.parse_to_grammar(f, grammar) 50 | except Exception as e: 51 | raise ParseError(f"Failed to parse grammar: {e}") 52 | 53 | grammar.validate() 54 | return grammar 55 | 56 | @classmethod 57 | def from_file(cls, file_path: Union[str, Path]) -> 'Grammar': 58 | """ 59 | Parse a grammar from a file. 60 | 61 | Args: 62 | file_path: Path to the JSGF grammar file 63 | 64 | Returns: 65 | A Grammar instance 66 | 67 | Raises: 68 | ParseError: If the grammar cannot be parsed 69 | FileNotFoundError: If the file doesn't exist 70 | """ 71 | path = Path(file_path) 72 | if not path.exists(): 73 | raise FileNotFoundError(f"Grammar file not found: {file_path}") 74 | 75 | try: 76 | with open(path, 'r', encoding='utf-8') as f: 77 | return cls.from_stream(f) 78 | except Exception as e: 79 | raise ParseError(f"Failed to parse grammar file {file_path}: {e}") 80 | 81 | @classmethod 82 | def from_stream(cls, stream: TextIO) -> 'Grammar': 83 | """ 84 | Parse a grammar from a text stream. 85 | 86 | Args: 87 | stream: Text stream containing JSGF grammar 88 | 89 | Returns: 90 | A Grammar instance 91 | 92 | Raises: 93 | ParseError: If the grammar cannot be parsed 94 | """ 95 | grammar = cls() 96 | adapter = LegacyAdapter() 97 | 98 | try: 99 | adapter.parse_to_grammar(stream, grammar) 100 | except Exception as e: 101 | raise ParseError(f"Failed to parse grammar: {e}") 102 | 103 | grammar.validate() 104 | return grammar 105 | 106 | def add_rule(self, rule: Rule) -> None: 107 | """ 108 | Add a rule to the grammar. 109 | 110 | Args: 111 | rule: The rule to add 112 | 113 | Raises: 114 | ValueError: If a rule with the same name already exists 115 | """ 116 | if rule.name in self._rules: 117 | raise ValueError(f"Rule '{rule.name}' already exists") 118 | 119 | self._rules[rule.name] = rule 120 | if rule.is_public: 121 | self._public_rules.append(rule) 122 | 123 | def get_rule(self, name: str) -> Optional[Rule]: 124 | """ 125 | Get a rule by name. 126 | 127 | Args: 128 | name: The rule name (with or without angle brackets) 129 | 130 | Returns: 131 | The rule if found, None otherwise 132 | """ 133 | # Handle both and name formats 134 | clean_name = name.strip('<>') 135 | return self._rules.get(f"<{clean_name}>") 136 | 137 | def has_rule(self, name: str) -> bool: 138 | """ 139 | Check if a rule exists. 140 | 141 | Args: 142 | name: The rule name (with or without angle brackets) 143 | 144 | Returns: 145 | True if the rule exists, False otherwise 146 | """ 147 | return self.get_rule(name) is not None 148 | 149 | @property 150 | def rules(self) -> Dict[str, Rule]: 151 | """Get all rules in the grammar.""" 152 | return self._rules.copy() 153 | 154 | @property 155 | def public_rules(self) -> List[Rule]: 156 | """Get all public rules in the grammar.""" 157 | return self._public_rules.copy() 158 | 159 | @property 160 | def rule_names(self) -> List[str]: 161 | """Get all rule names.""" 162 | return list(self._rules.keys()) 163 | 164 | @property 165 | def public_rule_names(self) -> List[str]: 166 | """Get all public rule names.""" 167 | return [rule.name for rule in self._public_rules] 168 | 169 | def validate(self) -> None: 170 | """ 171 | Validate the grammar for consistency and completeness. 172 | 173 | Raises: 174 | ValidationError: If the grammar is invalid 175 | """ 176 | errors = [] 177 | 178 | # Check that all referenced non-terminals have rules 179 | for rule in self._rules.values(): 180 | undefined_refs = self._find_undefined_references(rule.expansion) 181 | if undefined_refs: 182 | errors.append( 183 | f"Rule '{rule.name}' references undefined non-terminals: " 184 | f"{', '.join(undefined_refs)}" 185 | ) 186 | 187 | # Check for at least one public rule 188 | if not self._public_rules: 189 | errors.append("Grammar must have at least one public rule") 190 | 191 | if errors: 192 | raise ValidationError("Grammar validation failed:\n" + "\n".join(errors)) 193 | 194 | def _find_undefined_references(self, node: JSGFNode) -> List[str]: 195 | """Find all undefined non-terminal references in a node.""" 196 | undefined = [] 197 | 198 | def visit(n: JSGFNode): 199 | if isinstance(n, NonTerminal): 200 | if not self.has_rule(n.name): 201 | undefined.append(n.name) 202 | elif isinstance(n, Sequence): 203 | for element in n.elements: 204 | visit(element) 205 | elif isinstance(n, Alternative): 206 | for choice_node, _ in n.choices: 207 | visit(choice_node) 208 | elif isinstance(n, (OptionalNode, Group)): 209 | visit(n.element) 210 | 211 | visit(node) 212 | return undefined 213 | 214 | def detect_cycles(self) -> List[List[str]]: 215 | """ 216 | Detect cycles in the grammar rules. 217 | 218 | Returns: 219 | List of cycles, where each cycle is a list of rule names 220 | """ 221 | # Build dependency graph 222 | graph = {} 223 | for rule_name, rule in self._rules.items(): 224 | graph[rule_name] = self._get_direct_dependencies(rule.expansion) 225 | 226 | # Find strongly connected components (cycles) 227 | cycles = [] 228 | visited = set() 229 | rec_stack = set() 230 | 231 | def dfs(node: str, path: List[str]): 232 | if node in rec_stack: 233 | # Found a cycle 234 | cycle_start = path.index(node) 235 | cycle = path[cycle_start:] + [node] 236 | cycles.append(cycle) 237 | return 238 | 239 | if node in visited: 240 | return 241 | 242 | visited.add(node) 243 | rec_stack.add(node) 244 | path.append(node) 245 | 246 | for neighbor in graph.get(node, []): 247 | dfs(neighbor, path.copy()) 248 | 249 | rec_stack.remove(node) 250 | 251 | for rule_name in self._rules: 252 | if rule_name not in visited: 253 | dfs(rule_name, []) 254 | 255 | return cycles 256 | 257 | def _get_direct_dependencies(self, node: JSGFNode) -> List[str]: 258 | """Get direct non-terminal dependencies of a node.""" 259 | dependencies = [] 260 | 261 | def visit(n: JSGFNode): 262 | if isinstance(n, NonTerminal): 263 | dependencies.append(n.name) 264 | elif isinstance(n, Sequence): 265 | for element in n.elements: 266 | visit(element) 267 | elif isinstance(n, Alternative): 268 | for choice_node, _ in n.choices: 269 | visit(choice_node) 270 | elif isinstance(n, (OptionalNode, Group)): 271 | visit(n.element) 272 | 273 | visit(node) 274 | return dependencies 275 | 276 | def is_recursive(self, rule_name: Optional[str] = None) -> bool: 277 | """ 278 | Check if the grammar (or a specific rule) contains recursion. 279 | 280 | Args: 281 | rule_name: If provided, check if this specific rule is recursive. 282 | If None, check if any rule in the grammar is recursive. 283 | 284 | Returns: 285 | True if recursion is detected, False otherwise 286 | """ 287 | cycles = self.detect_cycles() 288 | 289 | if rule_name is None: 290 | return len(cycles) > 0 291 | 292 | # Check if the specific rule is involved in any cycle 293 | clean_name = rule_name.strip('<>') 294 | full_name = f"<{clean_name}>" 295 | 296 | for cycle in cycles: 297 | if full_name in cycle: 298 | return True 299 | 300 | return False 301 | 302 | def __str__(self) -> str: 303 | """Return a string representation of the grammar.""" 304 | lines = [] 305 | for rule in self._rules.values(): 306 | lines.append(str(rule)) 307 | return "\n".join(lines) 308 | 309 | def __len__(self) -> int: 310 | """Return the number of rules in the grammar.""" 311 | return len(self._rules) 312 | 313 | def __contains__(self, rule_name: str) -> bool: 314 | """Check if a rule name exists in the grammar.""" 315 | return self.has_rule(rule_name) 316 | 317 | def __iter__(self) -> Iterator[Rule]: 318 | """Iterate over all rules in the grammar.""" 319 | return iter(self._rules.values()) -------------------------------------------------------------------------------- /docs/_build/html/_static/basic.css: -------------------------------------------------------------------------------- 1 | /* 2 | * basic.css 3 | * ~~~~~~~~~ 4 | * 5 | * Sphinx stylesheet -- basic theme. 6 | * 7 | * :copyright: Copyright 2007-2014 by the Sphinx team, see AUTHORS. 8 | * :license: BSD, see LICENSE for details. 9 | * 10 | */ 11 | 12 | /* -- main layout ----------------------------------------------------------- */ 13 | 14 | div.clearer { 15 | clear: both; 16 | } 17 | 18 | /* -- relbar ---------------------------------------------------------------- */ 19 | 20 | div.related { 21 | width: 100%; 22 | font-size: 90%; 23 | } 24 | 25 | div.related h3 { 26 | display: none; 27 | } 28 | 29 | div.related ul { 30 | margin: 0; 31 | padding: 0 0 0 10px; 32 | list-style: none; 33 | } 34 | 35 | div.related li { 36 | display: inline; 37 | } 38 | 39 | div.related li.right { 40 | float: right; 41 | margin-right: 5px; 42 | } 43 | 44 | /* -- sidebar --------------------------------------------------------------- */ 45 | 46 | div.sphinxsidebarwrapper { 47 | padding: 10px 5px 0 10px; 48 | } 49 | 50 | div.sphinxsidebar { 51 | float: left; 52 | width: 230px; 53 | margin-left: -100%; 54 | font-size: 90%; 55 | } 56 | 57 | div.sphinxsidebar ul { 58 | list-style: none; 59 | } 60 | 61 | div.sphinxsidebar ul ul, 62 | div.sphinxsidebar ul.want-points { 63 | margin-left: 20px; 64 | list-style: square; 65 | } 66 | 67 | div.sphinxsidebar ul ul { 68 | margin-top: 0; 69 | margin-bottom: 0; 70 | } 71 | 72 | div.sphinxsidebar form { 73 | margin-top: 10px; 74 | } 75 | 76 | div.sphinxsidebar input { 77 | border: 1px solid #98dbcc; 78 | font-family: sans-serif; 79 | font-size: 1em; 80 | } 81 | 82 | div.sphinxsidebar #searchbox input[type="text"] { 83 | width: 170px; 84 | } 85 | 86 | div.sphinxsidebar #searchbox input[type="submit"] { 87 | width: 30px; 88 | } 89 | 90 | img { 91 | border: 0; 92 | max-width: 100%; 93 | } 94 | 95 | /* -- search page ----------------------------------------------------------- */ 96 | 97 | ul.search { 98 | margin: 10px 0 0 20px; 99 | padding: 0; 100 | } 101 | 102 | ul.search li { 103 | padding: 5px 0 5px 20px; 104 | background-image: url(file.png); 105 | background-repeat: no-repeat; 106 | background-position: 0 7px; 107 | } 108 | 109 | ul.search li a { 110 | font-weight: bold; 111 | } 112 | 113 | ul.search li div.context { 114 | color: #888; 115 | margin: 2px 0 0 30px; 116 | text-align: left; 117 | } 118 | 119 | ul.keywordmatches li.goodmatch a { 120 | font-weight: bold; 121 | } 122 | 123 | /* -- index page ------------------------------------------------------------ */ 124 | 125 | table.contentstable { 126 | width: 90%; 127 | } 128 | 129 | table.contentstable p.biglink { 130 | line-height: 150%; 131 | } 132 | 133 | a.biglink { 134 | font-size: 1.3em; 135 | } 136 | 137 | span.linkdescr { 138 | font-style: italic; 139 | padding-top: 5px; 140 | font-size: 90%; 141 | } 142 | 143 | /* -- general index --------------------------------------------------------- */ 144 | 145 | table.indextable { 146 | width: 100%; 147 | } 148 | 149 | table.indextable td { 150 | text-align: left; 151 | vertical-align: top; 152 | } 153 | 154 | table.indextable dl, table.indextable dd { 155 | margin-top: 0; 156 | margin-bottom: 0; 157 | } 158 | 159 | table.indextable tr.pcap { 160 | height: 10px; 161 | } 162 | 163 | table.indextable tr.cap { 164 | margin-top: 10px; 165 | background-color: #f2f2f2; 166 | } 167 | 168 | img.toggler { 169 | margin-right: 3px; 170 | margin-top: 3px; 171 | cursor: pointer; 172 | } 173 | 174 | div.modindex-jumpbox { 175 | border-top: 1px solid #ddd; 176 | border-bottom: 1px solid #ddd; 177 | margin: 1em 0 1em 0; 178 | padding: 0.4em; 179 | } 180 | 181 | div.genindex-jumpbox { 182 | border-top: 1px solid #ddd; 183 | border-bottom: 1px solid #ddd; 184 | margin: 1em 0 1em 0; 185 | padding: 0.4em; 186 | } 187 | 188 | /* -- general body styles --------------------------------------------------- */ 189 | 190 | a.headerlink { 191 | visibility: hidden; 192 | } 193 | 194 | h1:hover > a.headerlink, 195 | h2:hover > a.headerlink, 196 | h3:hover > a.headerlink, 197 | h4:hover > a.headerlink, 198 | h5:hover > a.headerlink, 199 | h6:hover > a.headerlink, 200 | dt:hover > a.headerlink { 201 | visibility: visible; 202 | } 203 | 204 | div.body p.caption { 205 | text-align: inherit; 206 | } 207 | 208 | div.body td { 209 | text-align: left; 210 | } 211 | 212 | .field-list ul { 213 | padding-left: 1em; 214 | } 215 | 216 | .first { 217 | margin-top: 0 !important; 218 | } 219 | 220 | p.rubric { 221 | margin-top: 30px; 222 | font-weight: bold; 223 | } 224 | 225 | img.align-left, .figure.align-left, object.align-left { 226 | clear: left; 227 | float: left; 228 | margin-right: 1em; 229 | } 230 | 231 | img.align-right, .figure.align-right, object.align-right { 232 | clear: right; 233 | float: right; 234 | margin-left: 1em; 235 | } 236 | 237 | img.align-center, .figure.align-center, object.align-center { 238 | display: block; 239 | margin-left: auto; 240 | margin-right: auto; 241 | } 242 | 243 | .align-left { 244 | text-align: left; 245 | } 246 | 247 | .align-center { 248 | text-align: center; 249 | } 250 | 251 | .align-right { 252 | text-align: right; 253 | } 254 | 255 | /* -- sidebars -------------------------------------------------------------- */ 256 | 257 | div.sidebar { 258 | margin: 0 0 0.5em 1em; 259 | border: 1px solid #ddb; 260 | padding: 7px 7px 0 7px; 261 | background-color: #ffe; 262 | width: 40%; 263 | float: right; 264 | } 265 | 266 | p.sidebar-title { 267 | font-weight: bold; 268 | } 269 | 270 | /* -- topics ---------------------------------------------------------------- */ 271 | 272 | div.topic { 273 | border: 1px solid #ccc; 274 | padding: 7px 7px 0 7px; 275 | margin: 10px 0 10px 0; 276 | } 277 | 278 | p.topic-title { 279 | font-size: 1.1em; 280 | font-weight: bold; 281 | margin-top: 10px; 282 | } 283 | 284 | /* -- admonitions ----------------------------------------------------------- */ 285 | 286 | div.admonition { 287 | margin-top: 10px; 288 | margin-bottom: 10px; 289 | padding: 7px; 290 | } 291 | 292 | div.admonition dt { 293 | font-weight: bold; 294 | } 295 | 296 | div.admonition dl { 297 | margin-bottom: 0; 298 | } 299 | 300 | p.admonition-title { 301 | margin: 0px 10px 5px 0px; 302 | font-weight: bold; 303 | } 304 | 305 | div.body p.centered { 306 | text-align: center; 307 | margin-top: 25px; 308 | } 309 | 310 | /* -- tables ---------------------------------------------------------------- */ 311 | 312 | table.docutils { 313 | border: 0; 314 | border-collapse: collapse; 315 | } 316 | 317 | table.docutils td, table.docutils th { 318 | padding: 1px 8px 1px 5px; 319 | border-top: 0; 320 | border-left: 0; 321 | border-right: 0; 322 | border-bottom: 1px solid #aaa; 323 | } 324 | 325 | table.field-list td, table.field-list th { 326 | border: 0 !important; 327 | } 328 | 329 | table.footnote td, table.footnote th { 330 | border: 0 !important; 331 | } 332 | 333 | th { 334 | text-align: left; 335 | padding-right: 5px; 336 | } 337 | 338 | table.citation { 339 | border-left: solid 1px gray; 340 | margin-left: 1px; 341 | } 342 | 343 | table.citation td { 344 | border-bottom: none; 345 | } 346 | 347 | /* -- other body styles ----------------------------------------------------- */ 348 | 349 | ol.arabic { 350 | list-style: decimal; 351 | } 352 | 353 | ol.loweralpha { 354 | list-style: lower-alpha; 355 | } 356 | 357 | ol.upperalpha { 358 | list-style: upper-alpha; 359 | } 360 | 361 | ol.lowerroman { 362 | list-style: lower-roman; 363 | } 364 | 365 | ol.upperroman { 366 | list-style: upper-roman; 367 | } 368 | 369 | dl { 370 | margin-bottom: 15px; 371 | } 372 | 373 | dd p { 374 | margin-top: 0px; 375 | } 376 | 377 | dd ul, dd table { 378 | margin-bottom: 10px; 379 | } 380 | 381 | dd { 382 | margin-top: 3px; 383 | margin-bottom: 10px; 384 | margin-left: 30px; 385 | } 386 | 387 | dt:target, .highlighted { 388 | background-color: #fbe54e; 389 | } 390 | 391 | dl.glossary dt { 392 | font-weight: bold; 393 | font-size: 1.1em; 394 | } 395 | 396 | .field-list ul { 397 | margin: 0; 398 | padding-left: 1em; 399 | } 400 | 401 | .field-list p { 402 | margin: 0; 403 | } 404 | 405 | .optional { 406 | font-size: 1.3em; 407 | } 408 | 409 | .versionmodified { 410 | font-style: italic; 411 | } 412 | 413 | .system-message { 414 | background-color: #fda; 415 | padding: 5px; 416 | border: 3px solid red; 417 | } 418 | 419 | .footnote:target { 420 | background-color: #ffa; 421 | } 422 | 423 | .line-block { 424 | display: block; 425 | margin-top: 1em; 426 | margin-bottom: 1em; 427 | } 428 | 429 | .line-block .line-block { 430 | margin-top: 0; 431 | margin-bottom: 0; 432 | margin-left: 1.5em; 433 | } 434 | 435 | .guilabel, .menuselection { 436 | font-family: sans-serif; 437 | } 438 | 439 | .accelerator { 440 | text-decoration: underline; 441 | } 442 | 443 | .classifier { 444 | font-style: oblique; 445 | } 446 | 447 | abbr, acronym { 448 | border-bottom: dotted 1px; 449 | cursor: help; 450 | } 451 | 452 | /* -- code displays --------------------------------------------------------- */ 453 | 454 | pre { 455 | overflow: auto; 456 | overflow-y: hidden; /* fixes display issues on Chrome browsers */ 457 | } 458 | 459 | td.linenos pre { 460 | padding: 5px 0px; 461 | border: 0; 462 | background-color: transparent; 463 | color: #aaa; 464 | } 465 | 466 | table.highlighttable { 467 | margin-left: 0.5em; 468 | } 469 | 470 | table.highlighttable td { 471 | padding: 0 0.5em 0 0.5em; 472 | } 473 | 474 | tt.descname { 475 | background-color: transparent; 476 | font-weight: bold; 477 | font-size: 1.2em; 478 | } 479 | 480 | tt.descclassname { 481 | background-color: transparent; 482 | } 483 | 484 | tt.xref, a tt { 485 | background-color: transparent; 486 | font-weight: bold; 487 | } 488 | 489 | h1 tt, h2 tt, h3 tt, h4 tt, h5 tt, h6 tt { 490 | background-color: transparent; 491 | } 492 | 493 | .viewcode-link { 494 | float: right; 495 | } 496 | 497 | .viewcode-back { 498 | float: right; 499 | font-family: sans-serif; 500 | } 501 | 502 | div.viewcode-block:target { 503 | margin: -1px -10px; 504 | padding: 0 10px; 505 | } 506 | 507 | /* -- math display ---------------------------------------------------------- */ 508 | 509 | img.math { 510 | vertical-align: middle; 511 | } 512 | 513 | div.body div.math p { 514 | text-align: center; 515 | } 516 | 517 | span.eqno { 518 | float: right; 519 | } 520 | 521 | /* -- printout stylesheet --------------------------------------------------- */ 522 | 523 | @media print { 524 | div.document, 525 | div.documentwrapper, 526 | div.bodywrapper { 527 | margin: 0 !important; 528 | width: 100%; 529 | } 530 | 531 | div.sphinxsidebar, 532 | div.related, 533 | div.footer, 534 | #top-link { 535 | display: none; 536 | } 537 | } -------------------------------------------------------------------------------- /jsgf/parser.py: -------------------------------------------------------------------------------- 1 | """ 2 | JSGF Grammar parser implementation. 3 | 4 | This module provides the JSGFParser class that converts JSGF grammar text 5 | into Grammar objects with proper AST representation. 6 | """ 7 | 8 | from typing import TextIO, List, Optional, Union, Any 9 | import re 10 | from pyparsing import ( 11 | Word, Literal, Group, Optional as PyparsingOptional, Forward, MatchFirst, 12 | Combine, alphas, alphanums, nums, stringEnd, ParseException, ParserElement, 13 | pyparsing_unicode 14 | ) 15 | 16 | from .ast_nodes import ( 17 | JSGFNode, Terminal, NonTerminal, Sequence, Alternative, 18 | Optional as OptionalNode, Group as GroupNode, Rule 19 | ) 20 | from .exceptions import ParseError 21 | 22 | 23 | # Enable packrat parsing for performance 24 | ParserElement.enablePackrat() 25 | 26 | # Unicode support: Tier 1 + Tier 2 scripts for comprehensive language coverage 27 | # Covers 5+ billion speakers: Latin, CJK, Arabic, Cyrillic, Devanagari, Hangul, Hebrew, Greek, Thai 28 | # Note: Using printables for scripts with combining characters (Thai, Devanagari) 29 | _unicode_letters = ( 30 | # Tier 1: Major scripts (Latin, CJK, Arabic, Cyrillic) 31 | pyparsing_unicode.Latin1.alphas + 32 | pyparsing_unicode.LatinA.alphas + 33 | pyparsing_unicode.LatinB.alphas + 34 | pyparsing_unicode.CJK.alphas + 35 | pyparsing_unicode.Arabic.alphas + 36 | pyparsing_unicode.Cyrillic.alphas + 37 | # Tier 2: Common scripts (using printables for scripts with combining marks) 38 | pyparsing_unicode.Devanagari.printables + 39 | pyparsing_unicode.Hangul.alphas + 40 | pyparsing_unicode.Hebrew.alphas + 41 | pyparsing_unicode.Greek.alphas + 42 | pyparsing_unicode.Thai.printables 43 | ) 44 | 45 | 46 | class JSGFParser: 47 | """ 48 | Parser for JSGF grammar files. 49 | 50 | This parser converts JSGF grammar text into a Grammar object containing 51 | properly structured AST nodes. 52 | """ 53 | 54 | def __init__(self): 55 | self._grammar_def = None 56 | self._setup_parser() 57 | 58 | def _setup_parser(self): 59 | """Set up the pyparsing grammar definition.""" 60 | 61 | # Basic tokens 62 | weight = ( 63 | Literal('/').suppress() + 64 | Word(nums + '.').setResultsName('weight_value') + 65 | Literal('/').suppress() 66 | ).setParseAction(self._parse_weight) 67 | 68 | token = ( 69 | Word(alphanums + _unicode_letters + "'_-,.?@!#$%^&*()+={}[]|\\:;\"~`") 70 | ).setParseAction(self._parse_token) 71 | 72 | nonterminal = ( 73 | Combine( 74 | Literal('<') + 75 | Word(alphanums + _unicode_letters + '$_:;,=|/\\()[]@#%!^&~') + 76 | Literal('>') 77 | ) 78 | ).setParseAction(self._parse_nonterminal) 79 | 80 | # Forward declarations for recursive grammar 81 | sequence = Forward() 82 | alternative = Forward() 83 | 84 | # Weighted expressions 85 | weighted_expr = ( 86 | weight + Group(sequence).setResultsName("expr") 87 | ).setParseAction(self._parse_weighted_expression) 88 | 89 | # Grouping and optional elements 90 | grouping = ( 91 | Literal('(').suppress() + 92 | alternative + 93 | Literal(')').suppress() 94 | ).setParseAction(self._parse_group) 95 | 96 | optional_grouping = ( 97 | Literal('[').suppress() + 98 | Group(alternative).setResultsName("optional_content") + 99 | Literal(']').suppress() 100 | ).setParseAction(self._parse_optional) 101 | 102 | # Basic expression elements 103 | expression = MatchFirst([ 104 | nonterminal, 105 | token, 106 | grouping, 107 | optional_grouping 108 | ]) 109 | 110 | # Sequence definition 111 | sequence <<= Group( 112 | expression + 113 | (expression)[...] 114 | ).setParseAction(self._parse_sequence) 115 | 116 | # Alternative definitions 117 | weighted_alternatives = Forward() 118 | weighted_prime = Literal('|').suppress() + weighted_alternatives 119 | weighted_alternatives <<= MatchFirst([ 120 | ( 121 | Group(weighted_expr).setResultsName("choice1") + 122 | Group(weighted_prime).setResultsName("choice2") 123 | ).setParseAction(self._parse_weighted_alternatives), 124 | Group(weighted_expr).setParseAction(self._parse_single_weighted) 125 | ]) 126 | 127 | regular_alternatives = Forward() 128 | regular_prime = Literal('|').suppress() + regular_alternatives 129 | regular_alternatives <<= MatchFirst([ 130 | ( 131 | Group(sequence).setResultsName("choice1") + 132 | Group(regular_prime).setResultsName("choice2") 133 | ).setParseAction(self._parse_regular_alternatives), 134 | Group(sequence).setParseAction(self._parse_single_regular) 135 | ]) 136 | 137 | # Top-level alternative 138 | alternative <<= MatchFirst([regular_alternatives, weighted_alternatives]) 139 | 140 | # Complete rule definition 141 | rule_def = ( 142 | PyparsingOptional(Literal('public')).setResultsName('is_public') + 143 | nonterminal.setResultsName('rule_name') + 144 | Literal('=').suppress() + 145 | Group(alternative).setResultsName('expansion') + 146 | Literal(';').suppress() 147 | ).setParseAction(self._parse_rule) 148 | 149 | self._grammar_def = rule_def 150 | 151 | def parse(self, stream: TextIO, grammar: 'Grammar') -> None: 152 | """ 153 | Parse a JSGF grammar from a text stream into a Grammar object. 154 | 155 | Args: 156 | stream: Text stream containing JSGF grammar 157 | grammar: Grammar object to populate with parsed rules 158 | 159 | Raises: 160 | ParseError: If parsing fails 161 | """ 162 | content = stream.read() 163 | 164 | # Remove comments 165 | content = self._remove_comments(content) 166 | 167 | # Split into individual rules and parse each one 168 | for line_num, line in enumerate(content.split('\n'), 1): 169 | line = line.strip() 170 | if not line: 171 | continue 172 | 173 | try: 174 | result = self._grammar_def.parseString(line, parseAll=True) 175 | rule = self._extract_rule(result) 176 | grammar.add_rule(rule) 177 | except ParseException as e: 178 | raise ParseError( 179 | f"Failed to parse rule: {str(e)}", 180 | line=line_num, 181 | column=e.column if hasattr(e, 'column') else None 182 | ) 183 | except Exception as e: 184 | raise ParseError(f"Unexpected error parsing rule: {str(e)}", line=line_num) 185 | 186 | def _remove_comments(self, text: str) -> str: 187 | """Remove comments from JSGF text.""" 188 | # Remove // style comments 189 | text = re.sub(r'//.*?$', '', text, flags=re.MULTILINE) 190 | # Remove /* */ style comments 191 | text = re.sub(r'/\*.*?\*/', '', text, flags=re.DOTALL) 192 | return text 193 | 194 | def _parse_weight(self, s: str, loc: int, tokens: Any) -> float: 195 | """Parse a weight value.""" 196 | return float(tokens.weight_value) 197 | 198 | def _parse_token(self, s: str, loc: int, tokens: Any) -> Terminal: 199 | """Parse a terminal token.""" 200 | return Terminal(tokens[0]) 201 | 202 | def _parse_nonterminal(self, s: str, loc: int, tokens: Any) -> NonTerminal: 203 | """Parse a non-terminal.""" 204 | return NonTerminal(tokens[0]) 205 | 206 | def _parse_sequence(self, s: str, loc: int, tokens: Any) -> Union[JSGFNode, Sequence]: 207 | """Parse a sequence of elements.""" 208 | elements = list(tokens[0]) 209 | if len(elements) == 1: 210 | return elements[0] 211 | return Sequence(elements) 212 | 213 | def _parse_group(self, s: str, loc: int, tokens: Any) -> GroupNode: 214 | """Parse a grouped expression.""" 215 | return GroupNode(tokens[0]) 216 | 217 | def _parse_optional(self, s: str, loc: int, tokens: Any) -> OptionalNode: 218 | """Parse an optional expression.""" 219 | return OptionalNode(tokens.optional_content[0]) 220 | 221 | def _parse_weighted_expression(self, s: str, loc: int, tokens: Any) -> tuple: 222 | """Parse a weighted expression.""" 223 | weight = tokens[0] # The weight value 224 | expr = tokens.expr[0] # The expression 225 | return (expr, weight) 226 | 227 | def _parse_weighted_alternatives(self, s: str, loc: int, tokens: Any) -> Alternative: 228 | """Parse weighted alternatives.""" 229 | choices = [] 230 | 231 | # Add first choice 232 | first_choice = tokens.choice1[0] 233 | if isinstance(first_choice, tuple): 234 | choices.append(first_choice) 235 | else: 236 | choices.append((first_choice, 1.0)) 237 | 238 | # Add remaining choices 239 | remaining = tokens.choice2[0] 240 | if isinstance(remaining, Alternative): 241 | choices.extend(remaining.choices) 242 | else: 243 | if isinstance(remaining, tuple): 244 | choices.append(remaining) 245 | else: 246 | choices.append((remaining, 1.0)) 247 | 248 | return Alternative(choices) 249 | 250 | def _parse_single_weighted(self, s: str, loc: int, tokens: Any) -> Alternative: 251 | """Parse a single weighted choice.""" 252 | choice = tokens[0] 253 | if isinstance(choice, tuple): 254 | return Alternative([choice]) 255 | else: 256 | return Alternative([(choice, 1.0)]) 257 | 258 | def _parse_regular_alternatives(self, s: str, loc: int, tokens: Any) -> Alternative: 259 | """Parse regular (unweighted) alternatives.""" 260 | choices = [] 261 | 262 | # Add first choice 263 | choices.append((tokens.choice1[0], 1.0)) 264 | 265 | # Add remaining choices 266 | remaining = tokens.choice2[0] 267 | if isinstance(remaining, Alternative): 268 | choices.extend(remaining.choices) 269 | else: 270 | choices.append((remaining, 1.0)) 271 | 272 | return Alternative(choices) 273 | 274 | def _parse_single_regular(self, s: str, loc: int, tokens: Any) -> Union[JSGFNode, Alternative]: 275 | """Parse a single regular choice.""" 276 | choice = tokens[0] 277 | # Don't wrap single elements in Alternative unnecessarily 278 | return choice 279 | 280 | def _parse_rule(self, s: str, loc: int, tokens: Any) -> dict: 281 | """Parse a complete rule definition.""" 282 | return { 283 | 'is_public': bool(tokens.is_public), 284 | 'name': tokens.rule_name.name, 285 | 'expansion': tokens.expansion[0] 286 | } 287 | 288 | def _extract_rule(self, parse_result: Any) -> Rule: 289 | """Extract a Rule object from parse results.""" 290 | return Rule( 291 | name=parse_result['name'], 292 | expansion=parse_result['expansion'], 293 | is_public=parse_result['is_public'] 294 | ) -------------------------------------------------------------------------------- /JSGFParser.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # @copyright: MIT License 3 | # Copyright (c) 2018 syntactic (Pastèque Ho) 4 | # Permission is hereby granted, free of charge, to any person obtaining a copy 5 | # of this software and associated documentation files (the "Software"), to deal 6 | # in the Software without restriction, including without limitation the rights 7 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 8 | # copies of the Software, and to permit persons to whom the Software is 9 | # furnished to do so, subject to the following conditions: 10 | # The above copyright notice and this permission notice shall be included in all 11 | # copies or substantial portions of the Software. 12 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 13 | # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 14 | # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 15 | # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 16 | # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 17 | # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 18 | # SOFTWARE. 19 | # @summary: This file parses a JSGF Grammar and prints it out. 20 | # @since: 2014/06/02 21 | 22 | """ 23 | This file parses a JSGF grammar file and returns a JSGFGrammar object. \ 24 | It uses the pyparsing module and defines a grammar for JSGF grammars. \ 25 | Upon finding a string or JSGF expression, it builds a grammar object from \ 26 | the bottom up, composing JSGF expressions with strings and lists. When the \ 27 | entire right hand side of a rule has been parsed and a JSGF expression \ 28 | object has been created of it, it gets added to the main JSGFGrammar \ 29 | object as one of its rules. 30 | 31 | To run the parser independently and print the resulting JSGFGrammar object, \ 32 | run it as: 33 | 34 | ``python JSGFParser.py Ideas.gram`` 35 | 36 | Generally, this module should be imported and the getGrammarObject should be called \ 37 | with a ``file`` object as its argument. This function returns a grammar \ 38 | object that can be used by the Generator scripts ``DeterministicGenerator.py`` \ 39 | and ``ProbabilisticGenerator.py``. 40 | 41 | The features of JSGF that this parser can handle include: 42 | - rulenames 43 | - tokens 44 | - comments 45 | - rule definitions 46 | - rule expansions 47 | - sequences 48 | - alternatives 49 | - weights 50 | - grouping 51 | - optional grouping 52 | 53 | Notable features of JSGF that are **not** handled by this parser are: 54 | - grammar names 55 | - import statements 56 | - unary operators 57 | - tags 58 | 59 | """ 60 | 61 | import sys 62 | import JSGFGrammar as gram 63 | from pyparsing import (Word, Literal, Group, Optional, ZeroOrMore, OneOrMore, 64 | Forward, MatchFirst, Combine, alphas, alphanums, nums, 65 | stringEnd, pyparsing_unicode) 66 | 67 | sys.setrecursionlimit(100000) 68 | usePackrat = True 69 | 70 | # Unicode support: Tier 1 + Tier 2 scripts for comprehensive language coverage 71 | # Covers 5+ billion speakers: Latin, CJK, Arabic, Cyrillic, Devanagari, Hangul, Hebrew, Greek, Thai 72 | # Note: Using printables for scripts with combining characters (Thai, Devanagari) 73 | _unicode_letters = ( 74 | # Tier 1: Major scripts (Latin, CJK, Arabic, Cyrillic) 75 | pyparsing_unicode.Latin1.alphas + 76 | pyparsing_unicode.LatinA.alphas + 77 | pyparsing_unicode.LatinB.alphas + 78 | pyparsing_unicode.CJK.alphas + 79 | pyparsing_unicode.Arabic.alphas + 80 | pyparsing_unicode.Cyrillic.alphas + 81 | # Tier 2: Common scripts (using printables for scripts with combining marks) 82 | pyparsing_unicode.Devanagari.printables + 83 | pyparsing_unicode.Hangul.alphas + 84 | pyparsing_unicode.Hebrew.alphas + 85 | pyparsing_unicode.Greek.alphas + 86 | pyparsing_unicode.Thai.printables 87 | ) 88 | 89 | def foundWeight(s, loc, toks): 90 | """ 91 | PyParsing action to run when a weight is found. 92 | 93 | :returns: Weight as a floating point number 94 | """ 95 | #print 'found weight', toks.dump() 96 | #print 'returning the weight', float(toks.weightAmount) 97 | return float(toks.weightAmount) 98 | 99 | def foundToken(s, loc, toks): 100 | """ 101 | PyParsing action to run when a token is found. 102 | 103 | :returns: Token as a string 104 | """ 105 | #print 'found token', toks.dump() 106 | #print 'returning the token', toks.token 107 | return toks.token 108 | 109 | def foundNonterminal(s, loc, toks): 110 | """ 111 | PyParsing action to run when a nonterminal reference is found. 112 | 113 | :returns: NonTerminal object representing the NT reference found 114 | """ 115 | return gram.NonTerminal(list(toks)[0]) 116 | 117 | def foundWeightedExpression(s, loc, toks): 118 | """ 119 | PyParsing action to run when a weighted expression is found. 120 | 121 | :returns: Ordered pair of the expression and its weight 122 | """ 123 | #print 'found weighted expression', toks.dump() 124 | expr = list(toks.expr) 125 | if len(expr) == 1: 126 | expr = expr[0] 127 | pair = (expr, toks.weight) 128 | #print 'returning', pair 129 | return pair 130 | 131 | def foundPair(s, loc, toks): 132 | """ 133 | PyParsing action to run when a pair of alternatives are found. 134 | 135 | :returns: Disjunction object containing all disjuncts that have been accumulated so far 136 | """ 137 | #print 'found pair', toks.dump() 138 | #print 'disj1 is', list(toks.disj1), 'disj2 is', list(toks.disj2) 139 | firstAlternative = list(toks.disj1) 140 | secondAlternative = list(toks.disj2) 141 | if len(firstAlternative) > 1: 142 | disj = [firstAlternative] 143 | else: 144 | disj = firstAlternative 145 | if len(secondAlternative) == 1: 146 | if isinstance(secondAlternative[0], gram.Disjunction): 147 | #print 'found disjuncts in second alt', secondAlternative[0].disjuncts 148 | disj.extend(secondAlternative[0].disjuncts) 149 | else: 150 | disj.append(secondAlternative[0]) 151 | else: 152 | disj.append(secondAlternative) 153 | disj = gram.Disjunction(disj) 154 | #print 'returing the pair', disj 155 | return disj 156 | 157 | def foundOptionalGroup(s, loc, toks): 158 | """ 159 | PyParsing action to run when an optional group is found. 160 | 161 | :returns: Optional object containing all elements in the group 162 | """ 163 | #print 'optional item is', toks.optionalItem 164 | if len(list(toks[0])) > 1: 165 | return gram.Optional(list(toks[0])) 166 | else: 167 | return gram.Optional(toks.optionalItem[0]) 168 | 169 | def foundSeq(s, loc, toks): 170 | """ 171 | PyParsing action to run when a sequence of concatenated elements is found. 172 | 173 | :returns: List of JSGFGrammar objects, strings, or more lists 174 | """ 175 | #print 'seq is', toks.dump() 176 | #print 'length of seq is', len(list(toks[0])), list(toks[0]) 177 | if len(list(toks[0])) > 1: 178 | #print 'seq retringin', list(toks[0]), type(list(toks[0])) 179 | return list(toks[0]) 180 | else: 181 | #print 'seq returning', list(toks[0])[0], type(list(toks[0])[0]) 182 | return list(toks[0])[0] 183 | 184 | # PyParsing rule for a weight 185 | weight = (Literal('/').suppress() + (Word(nums + '.')).setResultsName('weightAmount') + Literal('/').suppress()).setParseAction(foundWeight).setResultsName("weight") 186 | 187 | # PyParsing rule for a token (with Unicode support) 188 | token = Word(alphanums + _unicode_letters + "'_-,.?@").setResultsName('token').setParseAction(foundToken) 189 | 190 | # PyParsing rule for a nonterminal reference (with Unicode support) 191 | nonterminal = Combine(Literal('<') + Word(alphanums + _unicode_letters + '$_:;,=|/\\()[]@#%!^&~') + Literal('>')).setParseAction(foundNonterminal).setResultsName('NonTerminal') 192 | 193 | Sequence = Forward() 194 | 195 | weightedExpression = (weight + Group(Sequence).setResultsName("expr")).setResultsName('weightedExpression').setParseAction(foundWeightedExpression) 196 | 197 | weightAlternatives = Forward() 198 | weightedPrime = Literal('|').suppress() + weightAlternatives 199 | weightAlternatives << MatchFirst([(Group(weightedExpression).setResultsName("disj1") + Group(weightedPrime).setResultsName("disj2")).setParseAction(foundPair).setResultsName("pair"), Group(weightedExpression).setParseAction(foundSeq)]) 200 | 201 | disj = Forward() 202 | disjPrime = Literal('|').suppress() + disj 203 | disj << MatchFirst([(Group(Sequence).setResultsName("disj1") + Group(disjPrime).setResultsName("disj2")).setParseAction(foundPair).setResultsName("pair"), Group(Sequence).setParseAction(foundSeq)]) 204 | 205 | topLevel = MatchFirst([disj, weightAlternatives]) 206 | StartSymbol = Optional(Literal('public')).setResultsName('public') + nonterminal.setResultsName('identifier') + Literal('=').suppress() + Group(topLevel).setResultsName('ruleDef') + Literal(';').suppress() + stringEnd 207 | 208 | 209 | Expression = MatchFirst([nonterminal, token]) 210 | 211 | Grouping = Literal('(').suppress() + topLevel + Literal(')').suppress() 212 | OptionalGrouping = (Literal('[').suppress() + Group(topLevel).setResultsName("optionalItem") + Literal(']').suppress()).setParseAction(foundOptionalGroup) 213 | 214 | Sequence << Group(OneOrMore(MatchFirst([Grouping, OptionalGrouping, Expression]))).setResultsName("seq").setParseAction(foundSeq) 215 | 216 | def nocomment(oldline): 217 | """ 218 | Removes a comment from a line 219 | 220 | :param oldline: String representing the original line 221 | :returns: String with the same semantic content, with the comments stripped 222 | """ 223 | if '//' in oldline: 224 | return oldline.replace(oldline, oldline[0:oldline.index('//')]) 225 | elif '*' in oldline: 226 | return oldline.replace(oldline, '') 227 | else: 228 | return oldline 229 | 230 | def getGrammarObject(fileStream): 231 | """ 232 | Produces a JSGFGrammar object from a stream of text, the grammar object has a set of public rules and regular rules 233 | 234 | :param fileStream: file object containing the contents of the grammar file 235 | :type fileStream: file object 236 | :returns: JSGFGrammar object 237 | """ 238 | linegenerator = fileStream 239 | lines = linegenerator.readlines() 240 | for i in range(len(lines)): 241 | lines[i] = nocomment(lines[i]) 242 | # buffer will accumulate lines until a fully parseable piece is found 243 | buffer = "" 244 | 245 | grammar = gram.Grammar() 246 | for line in lines: 247 | buffer += line 248 | 249 | match = next(StartSymbol.scanString(buffer), None) 250 | while match: 251 | tokens, start, end = match 252 | #print 'rule dict is', tokens.asDict() 253 | if 'public' in tokens.keys(): 254 | grammar.addPublicRule(gram.Rule(tokens.identifier, list(tokens.ruleDef))) 255 | grammar.addRule(gram.Rule(tokens.identifier, list(tokens.ruleDef))) 256 | 257 | buffer = buffer[end:] 258 | match = next(StartSymbol.scanString(buffer), None) 259 | return grammar 260 | 261 | if __name__ == '__main__': 262 | fileStream = open(sys.argv[1]) 263 | grammar = getGrammarObject(fileStream) 264 | print(grammar) 265 | -------------------------------------------------------------------------------- /docs/_build/html/genindex.html: -------------------------------------------------------------------------------- 1 | 2 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | Index — JSGF Grammar Tools 1.0 documentation 11 | 12 | 13 | 14 | 15 | 24 | 25 | 26 | 27 | 28 | 29 | 30 |
31 |

Navigation

32 | 41 |
42 | 43 |
44 |
45 |
46 |
47 | 48 | 49 |

Index

50 | 51 |
52 | A 53 | | C 54 | | D 55 | | F 56 | | G 57 | | J 58 | | N 59 | | O 60 | | P 61 | | R 62 | | W 63 | 64 |
65 |

A

66 | 67 | 73 | 79 |
68 | 69 |
addPublicRule() (JSGFGrammar.Grammar method) 70 |
71 | 72 |
74 | 75 |
addRule() (JSGFGrammar.Grammar method) 76 |
77 | 78 |
80 | 81 |

C

82 | 83 | 95 |
84 | 85 |
combineSets() (in module DeterministicGenerator) 86 |
87 | 88 |
89 | 90 |
(in module ProbabilisticGenerator) 91 |
92 | 93 |
94 |
96 | 97 |

D

98 | 99 | 105 | 111 |
100 | 101 |
DeterministicGenerator (module) 102 |
103 | 104 |
106 | 107 |
Disjunction (class in JSGFGrammar) 108 |
109 | 110 |
112 | 113 |

F

114 | 115 | 133 | 147 |
116 | 117 |
foundNonterminal() (in module JSGFParser) 118 |
119 | 120 | 121 |
foundOptionalGroup() (in module JSGFParser) 122 |
123 | 124 | 125 |
foundPair() (in module JSGFParser) 126 |
127 | 128 | 129 |
foundSeq() (in module JSGFParser) 130 |
131 | 132 |
134 | 135 |
foundToken() (in module JSGFParser) 136 |
137 | 138 | 139 |
foundWeight() (in module JSGFParser) 140 |
141 | 142 | 143 |
foundWeightedExpression() (in module JSGFParser) 144 |
145 | 146 |
148 | 149 |

G

150 | 151 | 161 | 167 |
152 | 153 |
getGrammarObject() (in module JSGFParser) 154 |
155 | 156 | 157 |
getRHS() (JSGFGrammar.Grammar method) 158 |
159 | 160 |
162 | 163 |
Grammar (class in JSGFGrammar) 164 |
165 | 166 |
168 | 169 |

J

170 | 171 | 181 | 187 |
172 | 173 |
JSGFExpression (class in JSGFGrammar) 174 |
175 | 176 | 177 |
JSGFGrammar (module), [1] 178 |
179 | 180 |
182 | 183 |
JSGFParser (module) 184 |
185 | 186 |
188 | 189 |

N

190 | 191 | 197 | 203 |
192 | 193 |
nocomment() (in module JSGFParser) 194 |
195 | 196 |
198 | 199 |
NonTerminal (class in JSGFGrammar) 200 |
201 | 202 |
204 | 205 |

O

206 | 207 | 213 |
208 | 209 |
Optional (class in JSGFGrammar) 210 |
211 | 212 |
214 | 215 |

P

216 | 217 | 243 | 275 |
218 | 219 |
ProbabilisticGenerator (module) 220 |
221 | 222 | 223 |
processDisjunction() (in module DeterministicGenerator) 224 |
225 | 226 |
227 | 228 |
(in module ProbabilisticGenerator) 229 |
230 | 231 |
232 | 233 |
processNonTerminal() (in module DeterministicGenerator) 234 |
235 | 236 |
237 | 238 |
(in module ProbabilisticGenerator) 239 |
240 | 241 |
242 |
244 | 245 |
processOptional() (in module DeterministicGenerator) 246 |
247 | 248 |
249 | 250 |
(in module ProbabilisticGenerator) 251 |
252 | 253 |
254 | 255 |
processRHS() (in module DeterministicGenerator) 256 |
257 | 258 |
259 | 260 |
(in module ProbabilisticGenerator) 261 |
262 | 263 |
264 | 265 |
processSequence() (in module DeterministicGenerator) 266 |
267 | 268 |
269 | 270 |
(in module ProbabilisticGenerator) 271 |
272 | 273 |
274 |
276 | 277 |

R

278 | 279 | 285 |
280 | 281 |
Rule (class in JSGFGrammar) 282 |
283 | 284 |
286 | 287 |

W

288 | 289 | 295 |
290 | 291 |
weightedChoice() (in module ProbabilisticGenerator) 292 |
293 | 294 |
296 | 297 | 298 | 299 |
300 |
301 |
302 |
303 |
304 | 305 | 306 | 307 | 319 | 320 |
321 |
322 |
323 |
324 |
325 |

Navigation

326 | 335 |
336 |
337 | © Copyright 2014, Timothy Ho, Nicholas Bacuez. 338 | Created using Sphinx 1.2.2. 339 |
340 | 341 | -------------------------------------------------------------------------------- /docs/_build/html/_static/underscore.js: -------------------------------------------------------------------------------- 1 | // Underscore.js 1.3.1 2 | // (c) 2009-2012 Jeremy Ashkenas, DocumentCloud Inc. 3 | // Underscore is freely distributable under the MIT license. 4 | // Portions of Underscore are inspired or borrowed from Prototype, 5 | // Oliver Steele's Functional, and John Resig's Micro-Templating. 6 | // For all details and documentation: 7 | // http://documentcloud.github.com/underscore 8 | (function(){function q(a,c,d){if(a===c)return a!==0||1/a==1/c;if(a==null||c==null)return a===c;if(a._chain)a=a._wrapped;if(c._chain)c=c._wrapped;if(a.isEqual&&b.isFunction(a.isEqual))return a.isEqual(c);if(c.isEqual&&b.isFunction(c.isEqual))return c.isEqual(a);var e=l.call(a);if(e!=l.call(c))return false;switch(e){case "[object String]":return a==String(c);case "[object Number]":return a!=+a?c!=+c:a==0?1/a==1/c:a==+c;case "[object Date]":case "[object Boolean]":return+a==+c;case "[object RegExp]":return a.source== 9 | c.source&&a.global==c.global&&a.multiline==c.multiline&&a.ignoreCase==c.ignoreCase}if(typeof a!="object"||typeof c!="object")return false;for(var f=d.length;f--;)if(d[f]==a)return true;d.push(a);var f=0,g=true;if(e=="[object Array]"){if(f=a.length,g=f==c.length)for(;f--;)if(!(g=f in a==f in c&&q(a[f],c[f],d)))break}else{if("constructor"in a!="constructor"in c||a.constructor!=c.constructor)return false;for(var h in a)if(b.has(a,h)&&(f++,!(g=b.has(c,h)&&q(a[h],c[h],d))))break;if(g){for(h in c)if(b.has(c, 10 | h)&&!f--)break;g=!f}}d.pop();return g}var r=this,G=r._,n={},k=Array.prototype,o=Object.prototype,i=k.slice,H=k.unshift,l=o.toString,I=o.hasOwnProperty,w=k.forEach,x=k.map,y=k.reduce,z=k.reduceRight,A=k.filter,B=k.every,C=k.some,p=k.indexOf,D=k.lastIndexOf,o=Array.isArray,J=Object.keys,s=Function.prototype.bind,b=function(a){return new m(a)};if(typeof exports!=="undefined"){if(typeof module!=="undefined"&&module.exports)exports=module.exports=b;exports._=b}else r._=b;b.VERSION="1.3.1";var j=b.each= 11 | b.forEach=function(a,c,d){if(a!=null)if(w&&a.forEach===w)a.forEach(c,d);else if(a.length===+a.length)for(var e=0,f=a.length;e2;a== 12 | null&&(a=[]);if(y&&a.reduce===y)return e&&(c=b.bind(c,e)),f?a.reduce(c,d):a.reduce(c);j(a,function(a,b,i){f?d=c.call(e,d,a,b,i):(d=a,f=true)});if(!f)throw new TypeError("Reduce of empty array with no initial value");return d};b.reduceRight=b.foldr=function(a,c,d,e){var f=arguments.length>2;a==null&&(a=[]);if(z&&a.reduceRight===z)return e&&(c=b.bind(c,e)),f?a.reduceRight(c,d):a.reduceRight(c);var g=b.toArray(a).reverse();e&&!f&&(c=b.bind(c,e));return f?b.reduce(g,c,d,e):b.reduce(g,c)};b.find=b.detect= 13 | function(a,c,b){var e;E(a,function(a,g,h){if(c.call(b,a,g,h))return e=a,true});return e};b.filter=b.select=function(a,c,b){var e=[];if(a==null)return e;if(A&&a.filter===A)return a.filter(c,b);j(a,function(a,g,h){c.call(b,a,g,h)&&(e[e.length]=a)});return e};b.reject=function(a,c,b){var e=[];if(a==null)return e;j(a,function(a,g,h){c.call(b,a,g,h)||(e[e.length]=a)});return e};b.every=b.all=function(a,c,b){var e=true;if(a==null)return e;if(B&&a.every===B)return a.every(c,b);j(a,function(a,g,h){if(!(e= 14 | e&&c.call(b,a,g,h)))return n});return e};var E=b.some=b.any=function(a,c,d){c||(c=b.identity);var e=false;if(a==null)return e;if(C&&a.some===C)return a.some(c,d);j(a,function(a,b,h){if(e||(e=c.call(d,a,b,h)))return n});return!!e};b.include=b.contains=function(a,c){var b=false;if(a==null)return b;return p&&a.indexOf===p?a.indexOf(c)!=-1:b=E(a,function(a){return a===c})};b.invoke=function(a,c){var d=i.call(arguments,2);return b.map(a,function(a){return(b.isFunction(c)?c||a:a[c]).apply(a,d)})};b.pluck= 15 | function(a,c){return b.map(a,function(a){return a[c]})};b.max=function(a,c,d){if(!c&&b.isArray(a))return Math.max.apply(Math,a);if(!c&&b.isEmpty(a))return-Infinity;var e={computed:-Infinity};j(a,function(a,b,h){b=c?c.call(d,a,b,h):a;b>=e.computed&&(e={value:a,computed:b})});return e.value};b.min=function(a,c,d){if(!c&&b.isArray(a))return Math.min.apply(Math,a);if(!c&&b.isEmpty(a))return Infinity;var e={computed:Infinity};j(a,function(a,b,h){b=c?c.call(d,a,b,h):a;bd?1:0}),"value")};b.groupBy=function(a,c){var d={},e=b.isFunction(c)?c:function(a){return a[c]};j(a,function(a,b){var c=e(a,b);(d[c]||(d[c]=[])).push(a)});return d};b.sortedIndex=function(a, 17 | c,d){d||(d=b.identity);for(var e=0,f=a.length;e>1;d(a[g])=0})})};b.difference=function(a){var c=b.flatten(i.call(arguments,1));return b.filter(a,function(a){return!b.include(c,a)})};b.zip=function(){for(var a=i.call(arguments),c=b.max(b.pluck(a,"length")),d=Array(c),e=0;e=0;d--)b=[a[d].apply(this,b)];return b[0]}}; 24 | b.after=function(a,b){return a<=0?b():function(){if(--a<1)return b.apply(this,arguments)}};b.keys=J||function(a){if(a!==Object(a))throw new TypeError("Invalid object");var c=[],d;for(d in a)b.has(a,d)&&(c[c.length]=d);return c};b.values=function(a){return b.map(a,b.identity)};b.functions=b.methods=function(a){var c=[],d;for(d in a)b.isFunction(a[d])&&c.push(d);return c.sort()};b.extend=function(a){j(i.call(arguments,1),function(b){for(var d in b)a[d]=b[d]});return a};b.defaults=function(a){j(i.call(arguments, 25 | 1),function(b){for(var d in b)a[d]==null&&(a[d]=b[d])});return a};b.clone=function(a){return!b.isObject(a)?a:b.isArray(a)?a.slice():b.extend({},a)};b.tap=function(a,b){b(a);return a};b.isEqual=function(a,b){return q(a,b,[])};b.isEmpty=function(a){if(b.isArray(a)||b.isString(a))return a.length===0;for(var c in a)if(b.has(a,c))return false;return true};b.isElement=function(a){return!!(a&&a.nodeType==1)};b.isArray=o||function(a){return l.call(a)=="[object Array]"};b.isObject=function(a){return a===Object(a)}; 26 | b.isArguments=function(a){return l.call(a)=="[object Arguments]"};if(!b.isArguments(arguments))b.isArguments=function(a){return!(!a||!b.has(a,"callee"))};b.isFunction=function(a){return l.call(a)=="[object Function]"};b.isString=function(a){return l.call(a)=="[object String]"};b.isNumber=function(a){return l.call(a)=="[object Number]"};b.isNaN=function(a){return a!==a};b.isBoolean=function(a){return a===true||a===false||l.call(a)=="[object Boolean]"};b.isDate=function(a){return l.call(a)=="[object Date]"}; 27 | b.isRegExp=function(a){return l.call(a)=="[object RegExp]"};b.isNull=function(a){return a===null};b.isUndefined=function(a){return a===void 0};b.has=function(a,b){return I.call(a,b)};b.noConflict=function(){r._=G;return this};b.identity=function(a){return a};b.times=function(a,b,d){for(var e=0;e/g,">").replace(/"/g,""").replace(/'/g,"'").replace(/\//g,"/")};b.mixin=function(a){j(b.functions(a), 28 | function(c){K(c,b[c]=a[c])})};var L=0;b.uniqueId=function(a){var b=L++;return a?a+b:b};b.templateSettings={evaluate:/<%([\s\S]+?)%>/g,interpolate:/<%=([\s\S]+?)%>/g,escape:/<%-([\s\S]+?)%>/g};var t=/.^/,u=function(a){return a.replace(/\\\\/g,"\\").replace(/\\'/g,"'")};b.template=function(a,c){var d=b.templateSettings,d="var __p=[],print=function(){__p.push.apply(__p,arguments);};with(obj||{}){__p.push('"+a.replace(/\\/g,"\\\\").replace(/'/g,"\\'").replace(d.escape||t,function(a,b){return"',_.escape("+ 29 | u(b)+"),'"}).replace(d.interpolate||t,function(a,b){return"',"+u(b)+",'"}).replace(d.evaluate||t,function(a,b){return"');"+u(b).replace(/[\r\n\t]/g," ")+";__p.push('"}).replace(/\r/g,"\\r").replace(/\n/g,"\\n").replace(/\t/g,"\\t")+"');}return __p.join('');",e=new Function("obj","_",d);return c?e(c,b):function(a){return e.call(this,a,b)}};b.chain=function(a){return b(a).chain()};var m=function(a){this._wrapped=a};b.prototype=m.prototype;var v=function(a,c){return c?b(a).chain():a},K=function(a,c){m.prototype[a]= 30 | function(){var a=i.call(arguments);H.call(a,this._wrapped);return v(c.apply(b,a),this._chain)}};b.mixin(b);j("pop,push,reverse,shift,sort,splice,unshift".split(","),function(a){var b=k[a];m.prototype[a]=function(){var d=this._wrapped;b.apply(d,arguments);var e=d.length;(a=="shift"||a=="splice")&&e===0&&delete d[0];return v(d,this._chain)}});j(["concat","join","slice"],function(a){var b=k[a];m.prototype[a]=function(){return v(b.apply(this._wrapped,arguments),this._chain)}});m.prototype.chain=function(){this._chain= 31 | true;return this};m.prototype.value=function(){return this._wrapped}}).call(this); 32 | -------------------------------------------------------------------------------- /docs/_build/html/JSGFParser.html: -------------------------------------------------------------------------------- 1 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | JSGFParser module — JSGF Grammar Tools 1.0 documentation 10 | 11 | 12 | 13 | 14 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 |
32 |

Navigation

33 | 48 |
49 | 50 |
51 |
52 |
53 |
54 | 55 |
56 |

JSGFParser module

57 |

This file parses a JSGF grammar file and returns a JSGFGrammar object. It uses the pyparsing module and defines a grammar for JSGF grammars. Upon finding a string or JSGF expression, it builds a grammar object from the bottom up, composing JSGF expressions with strings and lists. When the entire right hand side of a rule has been parsed and a JSGF expression object has been created of it, it gets added to the main JSGFGrammar object as one of its rules.

58 |

To run the parser independently and print the resulting JSGFGrammar object, run it as:

59 |
60 |
python JSGFParser.py Ideas.gram
61 |

Generally, this module should be imported and the getGrammarObject should be called with a file object as its argument. This function returns a grammar object that can be used by the Generator scripts DeterministicGenerator.py and ProbabilisticGenerator.py.

62 |
63 |
The features of JSGF that this parser can handle include:
64 |
    65 |
  • rulenames
    • 66 |
  • tokens
    • 67 |
  • comments
    • 68 |
  • rule definitions
    • 69 |
  • rule expansions
    • 70 |
  • sequences
    • 71 |
  • alternatives
    • 72 |
  • weights
    • 73 |
  • grouping
    • 74 |
  • optional grouping
    • 75 |
76 |
77 |
Notable features of JSGF that are not handled by this parser are:
78 |
    79 |
  • grammar names
    • 80 |
  • import statements
    • 81 |
  • unary operators
    • 82 |
  • tags
    • 83 |
84 |
85 |
86 |
87 |
88 | JSGFParser.foundNonterminal(s, loc, toks)
89 |

PyParsing action to run when a nonterminal reference is found.

90 | 91 | 92 | 93 | 94 | 95 | 96 | 97 |
Returns:NonTerminal object representing the NT reference found
98 |
99 | 100 |
101 |
102 | JSGFParser.foundOptionalGroup(s, loc, toks)
103 |

PyParsing action to run when an optional group is found.

104 | 105 | 106 | 107 | 108 | 109 | 110 | 111 |
Returns:Optional object containing all elements in the group
112 |
113 | 114 |
115 |
116 | JSGFParser.foundPair(s, loc, toks)
117 |

PyParsing action to run when a pair of alternatives are found.

118 | 119 | 120 | 121 | 122 | 123 | 124 | 125 |
Returns:Disjunction object containing all disjuncts that have been accumulated so far
126 |
127 | 128 |
129 |
130 | JSGFParser.foundSeq(s, loc, toks)
131 |

PyParsing action to run when a sequence of concatenated elements is found.

132 | 133 | 134 | 135 | 136 | 137 | 138 | 139 |
Returns:List of JSGFGrammar objects, strings, or more lists
140 |
141 | 142 |
143 |
144 | JSGFParser.foundToken(s, loc, toks)
145 |

PyParsing action to run when a token is found.

146 | 147 | 148 | 149 | 150 | 151 | 152 | 153 |
Returns:Token as a string
154 |
155 | 156 |
157 |
158 | JSGFParser.foundWeight(s, loc, toks)
159 |

PyParsing action to run when a weight is found.

160 | 161 | 162 | 163 | 164 | 165 | 166 | 167 |
Returns:Weight as a floating point number
168 |
169 | 170 |
171 |
172 | JSGFParser.foundWeightedExpression(s, loc, toks)
173 |

PyParsing action to run when a weighted expression is found.

174 | 175 | 176 | 177 | 178 | 179 | 180 | 181 |
Returns:Ordered pair of the expression and its weight
182 |
183 | 184 |
185 |
186 | JSGFParser.getGrammarObject(fileStream)
187 |

Produces a JSGFGrammar object from a stream of text, the grammar object has a set of public rules and regular rules

188 | 189 | 190 | 191 | 192 | 193 | 194 | 195 | 196 | 197 |
Parameters:fileStream (file object) – file object containing the contents of the grammar file
Returns:JSGFGrammar object
198 |
199 | 200 |
201 |
202 | JSGFParser.nocomment(oldline)
203 |

Removes a comment from a line

204 | 205 | 206 | 207 | 208 | 209 | 210 | 211 | 212 | 213 |
Parameters:oldline – String representing the original line
Returns:String with the same semantic content, with the comments stripped
214 |
215 | 216 |
217 | 218 | 219 |
220 |
221 |
222 |
223 |
224 |

Previous topic

225 |

JSGFGrammar module

227 |

Next topic

228 |

Probabilistic Generator module

230 |

This Page

231 | 235 | 247 | 248 |
249 |
250 |
251 |
252 |
253 |

Navigation

254 | 269 |
270 |
271 | © Copyright 2014, Timothy Ho, Nicholas Bacuez. 272 | Created using Sphinx 1.2.2. 273 |
274 | 275 | --------------------------------------------------------------------------------