├── .env.example ├── .flake8 ├── .gitignore ├── .pre-commit-config.yaml ├── README.md ├── Untitled Diagram.drawio ├── __main__.py ├── config.json.example ├── data_extraction ├── __main__.py └── data_extractor.py ├── gpt-scraper.py ├── gpt_interaction ├── gpt_interaction.py ├── large_doc_processing.py └── qa_over_a_doc.ipynb ├── prd.md ├── requirements.txt ├── results └── denver.html ├── scraper_generation ├── __main__.py └── scraper_generator.py ├── tdd.md ├── tests └── test_gpt_interaction.py └── website_analysis ├── __init__.py ├── dom_analysis.py ├── experiments ├── proxy.py └── xhr.py └── utils.py /.env.example: -------------------------------------------------------------------------------- 1 | OPENAI_API_KEY= -------------------------------------------------------------------------------- /.flake8: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/djb-gt/gpt-automated-web-scraper/HEAD/.flake8 -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | venv 2 | config.json 3 | .env -------------------------------------------------------------------------------- /.pre-commit-config.yaml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/djb-gt/gpt-automated-web-scraper/HEAD/.pre-commit-config.yaml -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/djb-gt/gpt-automated-web-scraper/HEAD/README.md -------------------------------------------------------------------------------- /Untitled Diagram.drawio: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/djb-gt/gpt-automated-web-scraper/HEAD/Untitled Diagram.drawio -------------------------------------------------------------------------------- /__main__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /config.json.example: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/djb-gt/gpt-automated-web-scraper/HEAD/config.json.example -------------------------------------------------------------------------------- /data_extraction/__main__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /data_extraction/data_extractor.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/djb-gt/gpt-automated-web-scraper/HEAD/data_extraction/data_extractor.py -------------------------------------------------------------------------------- /gpt-scraper.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/djb-gt/gpt-automated-web-scraper/HEAD/gpt-scraper.py -------------------------------------------------------------------------------- /gpt_interaction/gpt_interaction.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/djb-gt/gpt-automated-web-scraper/HEAD/gpt_interaction/gpt_interaction.py -------------------------------------------------------------------------------- /gpt_interaction/large_doc_processing.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/djb-gt/gpt-automated-web-scraper/HEAD/gpt_interaction/large_doc_processing.py -------------------------------------------------------------------------------- /gpt_interaction/qa_over_a_doc.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/djb-gt/gpt-automated-web-scraper/HEAD/gpt_interaction/qa_over_a_doc.ipynb -------------------------------------------------------------------------------- /prd.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/djb-gt/gpt-automated-web-scraper/HEAD/prd.md -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/djb-gt/gpt-automated-web-scraper/HEAD/requirements.txt -------------------------------------------------------------------------------- /results/denver.html: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/djb-gt/gpt-automated-web-scraper/HEAD/results/denver.html -------------------------------------------------------------------------------- /scraper_generation/__main__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /scraper_generation/scraper_generator.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/djb-gt/gpt-automated-web-scraper/HEAD/scraper_generation/scraper_generator.py -------------------------------------------------------------------------------- /tdd.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/djb-gt/gpt-automated-web-scraper/HEAD/tdd.md -------------------------------------------------------------------------------- /tests/test_gpt_interaction.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/djb-gt/gpt-automated-web-scraper/HEAD/tests/test_gpt_interaction.py -------------------------------------------------------------------------------- /website_analysis/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/djb-gt/gpt-automated-web-scraper/HEAD/website_analysis/__init__.py -------------------------------------------------------------------------------- /website_analysis/dom_analysis.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/djb-gt/gpt-automated-web-scraper/HEAD/website_analysis/dom_analysis.py -------------------------------------------------------------------------------- /website_analysis/experiments/proxy.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/djb-gt/gpt-automated-web-scraper/HEAD/website_analysis/experiments/proxy.py -------------------------------------------------------------------------------- /website_analysis/experiments/xhr.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/djb-gt/gpt-automated-web-scraper/HEAD/website_analysis/experiments/xhr.py -------------------------------------------------------------------------------- /website_analysis/utils.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/djb-gt/gpt-automated-web-scraper/HEAD/website_analysis/utils.py --------------------------------------------------------------------------------