23 | 
24 | 
25 | 
26 | 
27 |
22 |
23 |
24 |
32 |
33 | ## :thinking: How it works:
34 | With CensusGPT, you can ask any question related to census data in natural language.
35 |
36 | These natural language questions get converted to SQL using GPT-3.5 and are then used to query the census database.
37 |
38 | Here are some examples:
39 |
40 | * [🔍 Five cities with a population over 100,000 and lowest crime](https://censusgpt.com/?s=five%20cities%20with%20a%20population%20over%20100%2C000%20and%20lowest%20crime)
41 | * [🔍 10 highest income areas in california](https://censusgpt.com/?s=10%20highest%20income%20areas%20in%20california)
42 |
43 | Here is a similar example from sfGPT:
44 |
45 | * [🔍 Which four neighborhoods had the most crime in San Francisco in 2021?](https://censusgpt.com/sf?s=Which+four+neighborhoods+had+the+most+crime+in+San+Francisco+in+2021%3F)
46 |
47 |
48 | #### Diagram:
49 |
50 | 
51 |
52 |
53 | ## :world_map: Roadmap:
54 |
55 | We're splitting the roadmap for this project broadly into two categories:
56 |
57 |
58 | ### 1. Visualizations:
59 |
60 | Currently, textSQL only supports visualizing zip codes and cities on an interactive map and bar chart using [Mapbox](https://www.mapbox.com/) + [Plotly](https://plotly.com/). But data can be visualized in other interesting ways such as Heatmaps and Pie charts. Not every kind of data can be (or should be) visualized on a map. For example, a query like _"What percent of total crime in San Francisco is burglary vs in New York City"_ is perfect for visualizing as a stacked bar chart, but really hard to visualize on map.
61 |
62 | Bar Chart:
63 |
64 | 
65 |
66 | [coming soon] Heatmap:
67 |
68 | 
69 |
70 | [coming soon] Visualization-GPT: A way to use natural language to create and iterate on data visualizations in natural language through a text-to-vega engine.
71 |
72 | ### 2. 🔌 Text-to-SQL BYOD (Bring Your Own Data) [here](https://github.com/caesarHQ/textSQL/tree/main/byod)
73 |
74 |
75 | You can now connect your own database & datasets to textSQL and self-host the service. Our vision is to continue to modularize and improve this process.
76 |
77 | #### Use cases
78 |
79 | - Public-facing interactive interfaces for data. Democratizing public data
80 | - Empowering researchers. Enabling journalists and other researchers to more easily explore data
81 | - Business intelligence. Reducing the burden on technical employees to build & run queries for non-technical
82 |
83 |
84 | Setup instructions for BYOD are [here](https://github.com/caesarHQ/textSQL/tree/main/byod).
85 |
86 |
87 | ## :pencil: Additional Notes
88 |
89 | #### Datasets:
90 |
91 | A lot of the users of this project have asked for additional data for both CensusGPT and sfGPT — historical census data (trends), weather, health, transportation and real-estate data. Feel free to create a pull request, drop a link to your dataset in our [Discord](https://discord.gg/JZtxhZQQus), or contribute data via our [dedicated submission form](https://airtable.com/shrDKRRGyRCihWEZd).
92 |
93 | More data → Better CensusGPT and sfGPT
94 |
95 | #### Query Building:
96 |
97 | Users build complex queries progressively. They start with a simple query like _"Which neighborhoods in LA have the best schools?"_ and then progressively add details like _"with median income that is under $100,000"_. One of the most powerful aspects of textSQL is enabling iterating on a query as a process of uncovering insights.
98 |
99 | ###
100 |
101 | ## :computer: How to Contribute:
102 |
103 | Join our [discord](https://discord.gg/JZtxhZQQus)
104 |
105 | ReadMe for the backend [here](https://github.com/caesarHQ/textSQL/blob/main/api/README.md)
106 |
107 | ReadMe for the frontend [here](https://github.com/caesarHQ/textSQL/blob/main/client/censusGPT/README.md)
108 |
109 |
110 | 
57 |
58 | #### Prerequisites
59 | `python3.10`
60 |
61 | #### Required configuration for development:
62 | - base URL for TextSQL API
63 |
64 | Configure the above in `.env`
65 |
66 | Example of `.env` file that should go in the following path `/byod/client`
67 | ```
68 | API_BASE="http://localhost:9000"
69 | ```
70 |
71 | When everything on localhost, this will point to the BYOD API on port 9000.
72 |
73 | #### Local development
74 |
75 | Initial setup
76 | ```
77 | $ ./scripts/setup.sh
78 | ```
79 |
80 | Activate virtual env
81 | ```
82 | $ source ./venv/bin/activate
83 | ```
84 |
85 | Run local instance
86 | ```
87 | $ ./scripts/dev.sh
88 | ```
89 |
90 | ## Facing issues? Got questions?
91 |
92 | Reach out in the discord for support: https://discord.com/invite/JZtxhZQQus
93 |
--------------------------------------------------------------------------------
/byod/api/.gitignore:
--------------------------------------------------------------------------------
1 | .env
2 | DS_Store
3 | venv/
4 | __pycache__/
5 | scratch/
--------------------------------------------------------------------------------
/byod/api/README.md:
--------------------------------------------------------------------------------
1 | # TextSQL API
2 |
3 | A Flask API for Text-to-SQL
4 |
5 | ## Prerequisites
6 | - `python3.10`
7 |
8 | ## Required configuration for development:
9 | - OpenAI Key
10 | - URL to the postgres DB
11 |
12 | Configure the above in `.env`. Refer to `.env.example`.
13 |
14 | ## Local development
15 |
16 | Initial setup
17 | ```sh
18 | $ ./scripts/setup.sh
19 | ```
20 |
21 | Activate virtual env
22 | ```sh
23 | $ source ./venv/bin/activate
24 | ```
25 |
26 | Run local instance
27 | ```sh
28 | $ ./scripts/dev.sh
29 | ```
30 |
--------------------------------------------------------------------------------
/byod/api/app/.env.example:
--------------------------------------------------------------------------------
1 | OPENAI_KEY=""
2 | PINECONE_KEY=""
3 | PINECONE_ENV=""
4 | DB_URL=""
5 | DB_MANAGED_METADATA="false"
--------------------------------------------------------------------------------
/byod/api/app/__init__.py:
--------------------------------------------------------------------------------
1 | from app.config import FlaskAppConfig, DB_MANAGED_METADATA
2 | from app.extensions import db
3 | # import models to create tables if they don't exist
4 | from app.models import in_context_examples, table_metadata, type_metadata
5 | from app.setup.routes import bp as setup_bp
6 | from app.sql_explanation.routes import bp as sql_explanation_bp
7 | from app.sql_generation.routes import bp as sql_gen_bp
8 | from app.table_selection.routes import bp as table_selection_bp
9 | from app.visualization.routes import bp as visualization_bp
10 | from app.table_selection.utils import load_tables_and_types_metadata
11 | from app.utils import load_in_context_examples
12 | from flask import Flask
13 | from flask_admin import Admin
14 | from flask_cors import CORS
15 | from flask_migrate import Migrate
16 |
17 |
18 | def create_app(config_object=FlaskAppConfig):
19 | app = Flask(__name__)
20 | app.config.from_object(config_object)
21 | CORS(app)
22 |
23 |
24 | # Initialize app with extensions
25 | db.init_app(app)
26 | migrate = Migrate(app, db)
27 | with app.app_context():
28 | if DB_MANAGED_METADATA:
29 | db.create_all()
30 | load_tables_and_types_metadata()
31 | load_in_context_examples()
32 | admin = Admin(None, name='admin', template_mode='bootstrap3')
33 | admin.init_app(app)
34 |
35 |
36 | @app.route("/ping")
37 | def ping():
38 | return 'pong'
39 |
40 | app.register_blueprint(setup_bp)
41 | app.register_blueprint(sql_explanation_bp)
42 | app.register_blueprint(sql_gen_bp)
43 | app.register_blueprint(table_selection_bp)
44 | app.register_blueprint(visualization_bp)
45 |
46 | # from app.errors import bp as errors_bp
47 | # app.register_blueprint(errors_bp)
48 |
49 | # from app.main import bp as main_bp
50 | # app.register_blueprint(main_bp)
51 |
52 | @app.teardown_request
53 | def session_clear(exception=None):
54 | db.session.remove()
55 | if exception:
56 | if db.session.is_active:
57 | db.session.rollback()
58 |
59 | return app
--------------------------------------------------------------------------------
/byod/api/app/config.py:
--------------------------------------------------------------------------------
1 | from os import getenv
2 |
3 | import openai
4 | import pinecone
5 | from dotenv import load_dotenv
6 | from sqlalchemy import create_engine
7 |
8 | load_dotenv()
9 |
10 | DB_URL = getenv("DB_URL")
11 | OPENAI_KEY = getenv("OPENAI_KEY")
12 | PINECONE_KEY = getenv("PINECONE_KEY")
13 | PINECONE_ENV = getenv("PINECONE_ENV")
14 | DB_MANAGED_METADATA = getenv("DB_MANAGED_METADATA")
15 | DB_MANAGED_METADATA= False if DB_MANAGED_METADATA is None else DB_MANAGED_METADATA.lower() == 'true'
16 |
17 | openai.api_key = OPENAI_KEY
18 |
19 | class FlaskAppConfig:
20 | CORS_HEADERS = "Content-Type"
21 | SQLALCHEMY_DATABASE_URI = DB_URL
22 | SQLALCHEMY_ENGINE_OPTIONS = {"pool_pre_ping": True}
23 |
24 | if DB_URL:
25 | ENGINE = create_engine(DB_URL)
26 | else:
27 | print('DB_URL not found, please check your environment')
28 | exit(1)
29 |
30 | if PINECONE_KEY and PINECONE_ENV:
31 | pinecone.init(
32 | api_key=PINECONE_KEY,
33 | environment=PINECONE_ENV
34 | )
--------------------------------------------------------------------------------
/byod/api/app/extensions.py:
--------------------------------------------------------------------------------
1 | from flask_sqlalchemy import SQLAlchemy
2 |
3 |
4 | db = SQLAlchemy()
--------------------------------------------------------------------------------
/byod/api/app/models/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/caesarHQ/textSQL/7393e19b1458f1004d312713f5fe49968449e33c/byod/api/app/models/__init__.py
--------------------------------------------------------------------------------
/byod/api/app/models/in_context_examples.py:
--------------------------------------------------------------------------------
1 | from dataclasses import dataclass
2 | from typing import Dict, List
3 |
4 | from app.extensions import db
5 |
6 | @dataclass
7 | class InContextExamples(db.Model):
8 | __tablename__ = "ai_sql_in_context_examples"
9 | mode = db.Column(db.String, primary_key=True)
10 | examples: List[Dict[str, str]] = db.Column(db.JSON)
--------------------------------------------------------------------------------
/byod/api/app/models/json/type_metadata.json:
--------------------------------------------------------------------------------
1 | {
2 | "sex": {
3 | "type": "sex",
4 | "valid_values": [
5 | "Both Sexes",
6 | "Female",
7 | "Male"
8 | ]
9 | },
10 | "race": {
11 | "type": "race",
12 | "valid_values": [
13 | "African American/Black",
14 | "All Races",
15 | "Asian",
16 | "Hispanic/Latino",
17 | "Multirace",
18 | "Native American",
19 | "Pacific Islander",
20 | "Some other race",
21 | "White"
22 | ]
23 | },
24 | "mortality_reasons": {
25 | "type": "mortality_reasons",
26 | "valid_values": [
27 | "All Cause Mortality",
28 | "All other and unspecified accidents and adverse effects",
29 | "All other diseases (residual)",
30 | "All other external causes",
31 | "Alzheimers disease",
32 | "Assault (homicide)",
33 | "Assault (homicide) by discharge of firearms",
34 | "Assault (homicide) by other and unspecified means",
35 | "Atherosclerosis",
36 | "Cerebrovascular diseases",
37 | "Certain conditions originating in the perinatal period",
38 | "Chronic liver disease and cirrhosis",
39 | "Chronic lower respiratory diseases",
40 | "Complications of medical and surgical care",
41 | "Congenital malformations",
42 | "Diabetes mellitus",
43 | "Essential hypertension and hypertensive renal disease",
44 | "Human immunodeficiency virus (HIV) disease",
45 | "Hypertensive heart disease with or without renal disease",
46 | "Influenza and pneumonia",
47 | "Intentional self-harm (suicide)",
48 | "Intentional self-harm (suicide) by discharge of firearms",
49 | "Intentional self-harm (suicide) by other and unspecified means",
50 | "Ischemic heart diseases",
51 | "Leukemia",
52 | "Malignant neoplasm of breast",
53 | "Malignant neoplasm of pancreas",
54 | "Malignant neoplasm of prostate",
55 | "Malignant neoplasm of stomach",
56 | "Malignant neoplasms of cervix uteri, corpus uteri and ovary",
57 | "Malignant neoplasms of colon, rectum and anus",
58 | "Malignant neoplasms of trachea, bronchus and lung",
59 | "Malignant neoplasms of urinary tract",
60 | "Motor vehicle accidents",
61 | "Nephritis, nephrotic syndrome and nephrosis",
62 | "Non-Hodgkins lymphoma",
63 | "Other diseases of circulatory system",
64 | "Other diseases of heart",
65 | "Other malignant neoplasms",
66 | "Peptic ulcer",
67 | "Pregnancy, childbirth and the puerperium",
68 | "Sudden infant death syndrome",
69 | "Other neonatal (excluding SIDS)",
70 | "Tuberculosis"
71 | ]
72 | },
73 | "Year": {
74 | "type": "Year",
75 | "valid_values": [
76 | "2012",
77 | "2013",
78 | "2014",
79 | "2015",
80 | "2016",
81 | "2017",
82 | "2018",
83 | "2019",
84 | "2020"
85 | ]
86 | }
87 | }
--------------------------------------------------------------------------------
/byod/api/app/models/table_metadata.py:
--------------------------------------------------------------------------------
1 | from dataclasses import dataclass
2 | from typing import Dict, List
3 |
4 | from app.extensions import db
5 |
6 | @dataclass
7 | class TableMetadata(db.Model):
8 | __tablename__ = "ai_sql_table_metadata"
9 | table_name = db.Column(db.String, primary_key=True)
10 | table_metadata: Dict[str, List[object]] = db.Column(db.JSON)
--------------------------------------------------------------------------------
/byod/api/app/models/type_metadata.py:
--------------------------------------------------------------------------------
1 | from dataclasses import dataclass
2 | from typing import Dict, List
3 |
4 | from app.extensions import db
5 |
6 | @dataclass
7 | class TypeMetadata(db.Model):
8 | __tablename__ = "ai_sql_type_metadata"
9 | type_name = db.Column(db.String, primary_key=True)
10 | type_metadata: Dict[str, List[object]] = db.Column(db.JSON)
--------------------------------------------------------------------------------
/byod/api/app/setup/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/caesarHQ/textSQL/7393e19b1458f1004d312713f5fe49968449e33c/byod/api/app/setup/__init__.py
--------------------------------------------------------------------------------
/byod/api/app/setup/routes.py:
--------------------------------------------------------------------------------
1 | from flask import Blueprint, jsonify, make_response, request
2 |
3 | from ..config import ENGINE
4 | from .utils import (ENUMS_METADATA_DICT, TABLES_METADATA_DICT,
5 | generate_few_shot_queries, generate_table_metadata,
6 | generate_type_metadata, get_table_names, get_type_names,
7 | save_table_metadata, save_type_metadata)
8 |
9 | bp = Blueprint('setup_bp', __name__)
10 |
11 | @bp.route('/setup', methods=['POST'])
12 | def setup_db():
13 | """
14 | Set up database for text to SQL
15 | """
16 | request_body = request.get_json()
17 | db_credentials = {}
18 | db_credentials["address"] = request_body.get("address")
19 | db_credentials["database"] = request_body.get("database")
20 | db_credentials["username"] = request_body.get("username")
21 | db_credentials["password"] = request_body.get("password")
22 | db_credentials["port"] = request_body.get("port", 5432)
23 |
24 | for key, value in db_credentials.items():
25 | if not value:
26 | error_msg = f"`{key}` is missing from request body"
27 | return make_response(jsonify({"error": error_msg}), 400)
28 |
29 |
30 | @bp.route('/tables', methods=['GET'])
31 | def get_tables():
32 | """
33 | Get table names from database
34 | """
35 | table_names = get_table_names()
36 | return make_response(jsonify({"table_names": table_names}), 200)
37 |
38 |
39 | @bp.route('/get_tables_metadata', methods=['POST'])
40 | def get_tables_metadata():
41 | """
42 | Get tables metadata
43 | """
44 | request_body = request.get_json()
45 | table_names = request_body.get('table_names')
46 |
47 | tables_metadata = {}
48 | for t in table_names:
49 | metadata = generate_table_metadata(t)
50 | tables_metadata[t] = metadata
51 |
52 | return make_response(jsonify({"tables_metadata": tables_metadata}), 200)
53 |
54 |
55 | @bp.route('/types', methods=['GET'])
56 | def get_types():
57 | """
58 | Get type names from database
59 | """
60 | type_names = get_type_names()
61 | return make_response(jsonify({"type_names": type_names}), 200)
62 |
63 |
64 | @bp.route('/get_types_metadata', methods=['POST'])
65 | def get_types_metadata():
66 | """
67 | Get types metadata
68 | """
69 | request_body = request.get_json()
70 | type_names = request_body.get('type_names')
71 |
72 | types_metadata = {}
73 | for t in type_names:
74 | metadata = generate_type_metadata(t)
75 | types_metadata[t] = metadata
76 |
77 | return make_response(jsonify({"types_metadata": types_metadata}), 200)
78 |
79 |
80 | @bp.route('/save_metadata', methods=['POST'])
81 | def save_metadata():
82 | request_body = request.get_json()
83 | tables_metadata_dict = request_body.get("tables_metadata_dict", {})
84 | types_metadata_dict = request_body.get("types_metadata_dict", {})
85 |
86 | for name, metadata in tables_metadata_dict.items():
87 | save_table_metadata(name, metadata)
88 |
89 | for name, metadata in types_metadata_dict.items():
90 | save_type_metadata(name, metadata)
91 |
92 | return "Success"
93 |
94 |
95 | # TODO: delete metadata
96 |
97 |
98 | # DEPRECATED
99 | @bp.route('/setup_metadata', methods=['POST'])
100 | def setup_metadata():
101 |
102 | # overwrite existing tables and enums metadata
103 | TABLES_METADATA_DICT = {}
104 | ENUMS_METADATA_DICT = {}
105 |
106 | for table_name in get_table_names():
107 | save_table_metadata(table_name, generate_table_metadata(table_name))
108 | for type_name in get_type_names():
109 | save_type_metadata(type_name, generate_type_metadata(type_name))
110 |
111 | return "Success"
--------------------------------------------------------------------------------
/byod/api/app/sql_explanation/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/caesarHQ/textSQL/7393e19b1458f1004d312713f5fe49968449e33c/byod/api/app/sql_explanation/__init__.py
--------------------------------------------------------------------------------
/byod/api/app/sql_explanation/routes.py:
--------------------------------------------------------------------------------
1 | from flask import Blueprint, jsonify, make_response, request
2 |
3 | from .utils import get_sql_explanation
4 |
5 | bp = Blueprint('sql_explanation_bp', __name__)
6 |
7 | @bp.route('/explain_sql', methods=['POST'])
8 | def get_tables():
9 | """
10 | Explains SQL in natural language
11 | """
12 | request_body = request.get_json()
13 | sql = request_body.get('sql')
14 |
15 | if not sql:
16 | error_msg = '`sql` is missing from request body'
17 | return make_response(jsonify({"error": error_msg}), 400)
18 |
19 | explanation = get_sql_explanation(sql)
20 | return make_response(jsonify({'explanation': explanation}), 200)
--------------------------------------------------------------------------------
/byod/api/app/sql_explanation/utils.py:
--------------------------------------------------------------------------------
1 | from ..utils import get_assistant_message, get_few_shot_messages
2 |
3 |
4 | def get_message_with_descriptions():
5 | message = (
6 | "Provide a concise explanation for the following SQL query: ```{sql}```"
7 | )
8 | return message
9 |
10 |
11 | def get_default_messages():
12 | default_messages = [{
13 | "role": "system",
14 | "content": "You are a helpful assistant for providing an explanation for a SQL query."
15 | }]
16 | default_messages.extend(get_few_shot_messages(mode="sql_explanation"))
17 | return default_messages
18 |
19 |
20 | def get_sql_explanation(sql) -> str:
21 | """
22 | Use language model to generate explanation of SQL query
23 | """
24 | content = get_message_with_descriptions().format(sql=sql)
25 | messages = get_default_messages().copy()
26 | messages.append({
27 | "role": "user",
28 | "content": content
29 | })
30 |
31 | model = "gpt-3.5-turbo"
32 |
33 | assistant_message_content = get_assistant_message(messages=messages, model=model)["message"]["content"]
34 | return assistant_message_content
--------------------------------------------------------------------------------
/byod/api/app/sql_generation/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/caesarHQ/textSQL/7393e19b1458f1004d312713f5fe49968449e33c/byod/api/app/sql_generation/__init__.py
--------------------------------------------------------------------------------
/byod/api/app/sql_generation/routes.py:
--------------------------------------------------------------------------------
1 | from flask import Blueprint, jsonify, make_response, request
2 |
3 | from ..config import PINECONE_ENV, PINECONE_KEY
4 | from ..table_selection.utils import (get_relevant_tables_from_lm,
5 | get_relevant_tables_from_pinecone)
6 | from .utils import text_to_sql_with_retry
7 |
8 | bp = Blueprint('sql_generation_bp', __name__)
9 |
10 |
11 | @bp.route('/text_to_sql', methods=['POST'])
12 | def text_to_sql():
13 | """
14 | Convert natural language query to SQL
15 | """
16 | request_body = request.get_json()
17 | natural_language_query = request_body.get("natural_language_query")
18 | table_names = request_body.get("table_names")
19 |
20 | if not natural_language_query:
21 | error_msg = "`natural_language_query` is missing from request body"
22 | return make_response(jsonify({"error": error_msg}), 400)
23 |
24 | try:
25 | if not table_names:
26 | if PINECONE_ENV and PINECONE_KEY:
27 | table_names = get_relevant_tables_from_pinecone(natural_language_query)
28 | else:
29 | table_names = get_relevant_tables_from_lm(natural_language_query)
30 | result, sql_query = text_to_sql_with_retry(natural_language_query, table_names)
31 | except Exception as e:
32 | error_msg = f"Error processing request: {str(e)}"
33 | return make_response(jsonify({"error": error_msg}), 500)
34 |
35 | return make_response(jsonify({"result": result, "sql_query": sql_query}), 200)
36 |
--------------------------------------------------------------------------------
/byod/api/app/sql_generation/utils.py:
--------------------------------------------------------------------------------
1 | from collections import OrderedDict
2 | from typing import Dict, List
3 |
4 | from app.config import ENGINE
5 | from sqlalchemy import text
6 |
7 | from ..table_selection.utils import get_table_schemas_str
8 | from ..utils import (extract_sql_query_from_message, get_assistant_message,
9 | get_few_shot_messages)
10 |
11 | MSG_WITH_ERROR_TRY_AGAIN = (
12 | """
13 | Try again.
14 | Only respond with valid SQL. Make sure to write your answer in markdown format.
15 | The SQL query you just generated resulted in the following error message:
16 | ---------------------
17 | {error_message}
18 | ---------------------
19 | """
20 | )
21 |
22 |
23 | def make_default_messages(schemas_str: str) -> List[Dict[str, str]]:
24 | # default_messages = [{
25 | # "role": "system",
26 | # "content": (
27 | # f"""
28 | # You are a helpful assistant for generating syntactically correct read-only SQL to answer a given question or command.
29 | # The following are tables you can query:
30 | # ---------------------
31 | # {schemas_str}
32 | # ---------------------
33 | # Make sure to write your answer in markdown format.
34 | # """
35 | # # TODO: place warnings here
36 | # # i.e. "Make sure each value in the result table is not null."
37 | # )
38 | # }]
39 | default_messages = []
40 | default_messages.extend(get_few_shot_messages(mode="text_to_sql"))
41 | return default_messages
42 |
43 |
44 | def make_rephrase_msg_with_schema_and_warnings():
45 | return (
46 | """
47 | Let's start by fixing and rephrasing the query to be more analytical. Use the schema context to rephrase the user question in a way that leads to optimal query results: {natural_language_query}
48 | The following are schemas of tables you can query:
49 | ---------------------
50 | {schemas_str}
51 | ---------------------
52 | Do not include any of the table names in the query.
53 | Ask the natural language query the way a data analyst, with knowledge of these tables, would.
54 | """
55 | )
56 |
57 | def make_msg_with_schema_and_warnings():
58 | return (
59 | """
60 | Generate syntactically correct read-only SQL to answer the following question/command: {natural_language_query}
61 | The following are schemas of tables you can query:
62 | ---------------------
63 | {schemas_str}
64 | ---------------------
65 | Make sure to write your answer in markdown format.
66 | """
67 | # TODO: place warnings here
68 | # i.e. "Make sure each value in the result table is not null.""
69 | )
70 |
71 | def is_read_only_query(sql_query: str) -> bool:
72 | """
73 | Checks if the given SQL query string is read-only.
74 | Returns True if the query is read-only, False otherwise.
75 | """
76 | # List of SQL statements that modify data in the database
77 | modifying_statements = ["INSERT", "UPDATE", "DELETE", "DROP", "CREATE", "ALTER", "GRANT", "TRUNCATE", "LOCK TABLES", "UNLOCK TABLES"]
78 |
79 | # Check if the query contains any modifying statements
80 | for statement in modifying_statements:
81 | if statement in sql_query.upper():
82 | return False
83 |
84 | # If no modifying statements are found, the query is read-only
85 | return True
86 |
87 |
88 | class NotReadOnlyException(Exception):
89 | pass
90 |
91 |
92 | class NullValueException(Exception):
93 | pass
94 |
95 |
96 | def execute_sql(sql_query: str):
97 | if not is_read_only_query(sql_query):
98 | raise NotReadOnlyException("Only read-only queries are allowed.")
99 |
100 | with ENGINE.connect() as connection:
101 | connection = connection.execution_options(postgresql_readonly=True)
102 | with connection.begin():
103 | result = connection.execute(text(sql_query))
104 |
105 | column_names = list(result.keys())
106 |
107 | rows = [list(r) for r in result.all()]
108 |
109 | # Check for null values
110 | # for row in rows:
111 | # for value in row:
112 | # if value is None:
113 | # raise NullValueException("Make sure each value in the result table is not null.")
114 |
115 |
116 | results = []
117 | for row in rows:
118 | result = OrderedDict()
119 | for i, column_name in enumerate(column_names):
120 | result[column_name] = row[i]
121 | results.append(result)
122 |
123 | result_dict = {
124 | "column_names": column_names,
125 | "results": results,
126 | }
127 | if results:
128 | result_dict["column_types"] = [type(r).__name__ for r in results[0]]
129 |
130 | return result_dict
131 |
132 |
133 | def text_to_sql_with_retry(natural_language_query, table_names, k=3, messages=None):
134 | """
135 | Tries to take a natural language query and generate valid SQL to answer it K times
136 | """
137 | if not messages:
138 | # ask the assistant to rephrase before generating the query
139 | schemas_str = get_table_schemas_str(table_names)
140 | # rephrase = [{
141 | # "role": "user",
142 | # "content": make_rephrase_msg_with_schema_and_warnings().format(
143 | # natural_language_query=natural_language_query,
144 | # schemas_str=schemas_str
145 | # )
146 | # }]
147 | # rephrased_query = get_assistant_message(rephrase)["message"]["content"]
148 | # natural_language_query = rephrased_query
149 |
150 | content = make_msg_with_schema_and_warnings().format(
151 | natural_language_query=natural_language_query,
152 | schemas_str=schemas_str
153 | )
154 |
155 | messages = make_default_messages(schemas_str)
156 | messages.append({
157 | "role": "user",
158 | "content": content
159 | })
160 |
161 | assistant_message = None
162 |
163 | for _ in range(k):
164 | try:
165 | # model = "gpt-4"
166 | # model = "gpt-3.5-turbo"
167 | model = "gpt-3.5-turbo-0301"
168 | assistant_message = get_assistant_message(messages, model=model)
169 | sql_query = extract_sql_query_from_message(assistant_message["message"]["content"])
170 |
171 | response = execute_sql(sql_query)
172 | # Generated SQL query did not produce exception. Return result
173 | return response, sql_query
174 |
175 | except Exception as e:
176 | messages.append({
177 | "role": "assistant",
178 | "content": assistant_message["message"]["content"]
179 | })
180 | messages.append({
181 | "role": "user",
182 | "content": MSG_WITH_ERROR_TRY_AGAIN.format(error_message=str(e))
183 | })
184 |
185 | print("Could not generate SQL query after {k} tries.".format(k=k))
186 | return None, None
187 |
--------------------------------------------------------------------------------
/byod/api/app/table_selection/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/caesarHQ/textSQL/7393e19b1458f1004d312713f5fe49968449e33c/byod/api/app/table_selection/__init__.py
--------------------------------------------------------------------------------
/byod/api/app/table_selection/routes.py:
--------------------------------------------------------------------------------
1 | from flask import Blueprint, jsonify, make_response, request
2 |
3 | from .utils import get_relevant_tables_from_pinecone
4 |
5 | bp = Blueprint('table_selection_bp', __name__)
6 |
7 | @bp.route('/get_tables', methods=['POST'])
8 | def get_tables():
9 | """
10 | Select relevant tables given a natural language query
11 | """
12 | request_body = request.get_json()
13 | natural_language_query = request_body.get("natural_language_query")
14 |
15 | if not natural_language_query:
16 | error_msg = '`natural_language_query` is missing from request body'
17 | return make_response(jsonify({"error": error_msg}), 400)
18 |
19 | table_names = get_relevant_tables_from_pinecone(natural_language_query)
20 | return make_response(jsonify({"table_names": table_names}), 200)
--------------------------------------------------------------------------------
/byod/api/app/table_selection/utils.py:
--------------------------------------------------------------------------------
1 | import json
2 | import re
3 | from typing import List
4 |
5 | import pinecone
6 | from app.config import DB_MANAGED_METADATA
7 | from app.extensions import db
8 | from app.models.table_metadata import TableMetadata
9 | from app.models.type_metadata import TypeMetadata
10 | from openai.embeddings_utils import get_embedding
11 |
12 | from ..utils import get_assistant_message, get_few_shot_messages
13 |
14 | ENUMS_METADATA_DICT = {}
15 | TABLES_METADATA_DICT = {}
16 | def load_tables_and_types_metadata():
17 | """
18 | Setup metadata dicts for tables and enums
19 | """
20 | global ENUMS_METADATA_DICT
21 | global TABLES_METADATA_DICT
22 |
23 | if not DB_MANAGED_METADATA:
24 | with open("app/models/json/table_metadata.json", "r") as f:
25 | TABLES_METADATA_DICT = json.load(f)
26 | with open("app/models/json/type_metadata.json", "r") as f:
27 | ENUMS_METADATA_DICT = json.load(f)
28 | return
29 |
30 | try:
31 | enums_metadata = TypeMetadata.query.all()
32 | except Exception as e:
33 | print(e)
34 | enums_metadata = []
35 | for enum_metadata in enums_metadata:
36 | # ENUMS_METADATA_DICT[enum_metadata.type_name] = enum_metadata
37 | ENUMS_METADATA_DICT[enum_metadata.type_name] = enum_metadata.type_metadata
38 |
39 | try:
40 | tables_metadata = TableMetadata.query.all()
41 | except Exception as e:
42 | print(e)
43 | tables_metadata = []
44 | for table_metadata in tables_metadata:
45 | # TABLES_METADATA_DICT[table_metadata.table_name] = table_metadata
46 | TABLES_METADATA_DICT[table_metadata.table_name] = table_metadata.table_metadata
47 |
48 |
49 |
50 | def save_tables_metadata_to_json():
51 | with open("app/models/json/table_metadata.json", "w") as f:
52 | json.dump(TABLES_METADATA_DICT, f, indent=4)
53 |
54 |
55 | def save_enums_metadata_to_json():
56 | with open("app/models/json/type_metadata.json", "w") as f:
57 | json.dump(ENUMS_METADATA_DICT, f, indent=4)
58 |
59 | # # TODO: load few shot from json
60 | # def save_few_shots_to_json():
61 | # with open("app/models/json/in_context_examples.json", "w") as f:
62 | # json.dump(IN_CONTEXT_EXAMPLES_DICT, f, indent=4)
63 |
64 |
65 | # TODO: refac this to access JSON fields instead of tables
66 | def get_table_schemas_str(table_names: List[str] = []) -> str:
67 | """
68 | Format table and types metadata into string to be used in prompt
69 | """
70 | global ENUMS_METADATA_DICT
71 | global TABLES_METADATA_DICT
72 |
73 | tables_to_use = []
74 | if table_names:
75 | tables_to_use = [TABLES_METADATA_DICT[t_name] for t_name in table_names]
76 | else:
77 | tables_to_use = [t for t in TABLES_METADATA_DICT.values()]
78 |
79 | enums_to_use = set()
80 | tables_str_list = []
81 | for table in tables_to_use:
82 | tables_str = f"table name: {table['name']}\n"
83 | if table.get("description"):
84 | tables_str += f"table description: {table.get('description')}\n"
85 | columns_str_list = []
86 | for column in table.get("columns", []):
87 | columns_str_list.append(f"{column['name']} [{column['type']}]")
88 | if column.get("type") in ENUMS_METADATA_DICT.keys():
89 | enums_to_use.add(column.get("type"))
90 | tables_str += f"table columns: {', '.join(columns_str_list)}\n"
91 | tables_str_list.append(tables_str)
92 | tables_details = "\n\n".join(tables_str_list)
93 |
94 | enums_str_list = []
95 | for custom_type_str in enums_to_use:
96 | custom_type = ENUMS_METADATA_DICT.get(custom_type_str)
97 | if custom_type:
98 | enums_str = f"enum: {custom_type['type']}\n"
99 | enums_str += f"valid values: {', '.join(custom_type.get('valid_values'))}\n"
100 | enums_str_list.append(enums_str)
101 | enums_details = "\n\n".join(enums_str_list)
102 |
103 | return enums_details + "\n\n" + tables_details
104 |
105 |
106 | def get_relevant_tables_from_pinecone(natural_language_query, index_name="text_to_sql") -> List[str]:
107 | """
108 | Identify relevant tables for answering a natural language query via vector store
109 | """
110 | vector = get_embedding(natural_language_query, "text-embedding-ada-002")
111 |
112 | results = pinecone.Index(index_name).query(
113 | vector=vector,
114 | top_k=5,
115 | include_metadata=True,
116 | )
117 |
118 | table_names = set()
119 | for result in results["matches"]:
120 | for table_name in result.metadata["table_names"]:
121 | table_names.add(table_name)
122 |
123 | print(results["matches"])
124 |
125 | return list(table_names)
126 |
127 |
128 | def _get_table_selection_message_with_descriptions(natural_language_query):
129 | return f"""
130 | Return a JSON object with relevant SQL tables for answering the following natural language query:
131 | ---------------------
132 | {natural_language_query}
133 | ---------------------
134 | Respond in JSON format with your answer in a field named \"tables\" which is a list of strings.
135 | Respond with an empty list if you cannot identify any relevant tables.
136 | Make sure to write your answer in markdown format.
137 | The following are descriptions of available tables and enums:
138 | ---------------------
139 | {get_table_schemas_str()}
140 | ---------------------
141 | """
142 |
143 |
144 | def _get_table_selection_messages():
145 | # default_messages = [{
146 | # "role": "system",
147 | # "content": (
148 | # f"""
149 | # You are a helpful assistant for identifying relevant SQL tables to use for answering a natural language query.
150 | # You respond in JSON format with your answer in a field named \"tables\" which is a list of strings.
151 | # Respond with an empty list if you cannot identify any relevant tables.
152 | # Make sure to write your answer in markdown format.
153 | # The following are descriptions of available tables and enums:
154 | # ---------------------
155 | # {get_table_schemas_str()}
156 | # ---------------------
157 | # """
158 | # )
159 | # }]
160 | default_messages = []
161 | default_messages.extend(get_few_shot_messages(mode="table_selection"))
162 | return default_messages
163 |
164 |
165 | def _extract_text_from_markdown(text):
166 | matches = re.findall(r"```([\s\S]+?)```", text)
167 | if matches:
168 | return matches[0]
169 | return text
170 |
171 |
172 | def get_relevant_tables_from_lm(natural_language_query):
173 | """
174 | Identify relevant tables for answering a natural language query via LM
175 | """
176 | content = _get_table_selection_message_with_descriptions(natural_language_query)
177 | messages = _get_table_selection_messages().copy()
178 | messages.append({
179 | "role": "user",
180 | "content": content
181 | })
182 |
183 | tables_json_str = _extract_text_from_markdown(
184 | get_assistant_message(
185 | messages=messages,
186 | model="gpt-3.5-turbo-0301",
187 | )["message"]["content"]
188 | )
189 | tables = json.loads(tables_json_str).get("tables")
190 | return tables
--------------------------------------------------------------------------------
/byod/api/app/utils.py:
--------------------------------------------------------------------------------
1 | import json
2 | import re
3 | from typing import Dict, List
4 |
5 | import openai
6 | from app.config import DB_MANAGED_METADATA
7 | from app.extensions import db
8 | from app.models.in_context_examples import InContextExamples
9 |
10 | IN_CONTEXT_EXAMPLES_DICT = {}
11 | def load_in_context_examples():
12 | """
13 | Setup in context examples dict
14 | """
15 | global IN_CONTEXT_EXAMPLES_DICT
16 |
17 | if not DB_MANAGED_METADATA:
18 | with open("app/models/json/in_context_examples.json", "r") as f:
19 | IN_CONTEXT_EXAMPLES_DICT = json.load(f)
20 | return
21 |
22 | try:
23 | in_context_examples = InContextExamples.query.all()
24 | except Exception as e:
25 | print(e)
26 | in_context_examples = []
27 | for in_context_example in in_context_examples:
28 | IN_CONTEXT_EXAMPLES_DICT[in_context_example.mode] = in_context_example.examples
29 |
30 |
31 | def get_few_shot_messages(mode: str = "text_to_sql") -> List[Dict]:
32 | global IN_CONTEXT_EXAMPLES_DICT
33 |
34 | examples = IN_CONTEXT_EXAMPLES_DICT.get(mode, [])
35 | messages = []
36 | for example in examples:
37 | messages.append({
38 | "role": "user",
39 | "content": example["user"],
40 | })
41 | messages.append({
42 | "role": "assistant",
43 | "content": example["assistant"],
44 | })
45 | return messages
46 |
47 |
48 | def get_assistant_message(
49 | messages: List[Dict[str, str]],
50 | temperature: int = 0,
51 | model: str = "gpt-3.5-0301",
52 | # model: str = "gpt-3.5-turbo",
53 | # model: str = "gpt-4",
54 | ):
55 | res = openai.ChatCompletion.create(
56 | model=model,
57 | temperature=temperature,
58 | messages=messages
59 | )
60 | # completion = res['choices'][0]["message"]["content"]
61 | assistant_message = res['choices'][0]
62 | return assistant_message
63 |
64 |
65 | def clean_message_content(assistant_message_content):
66 | """
67 | Cleans message content to extract the SQL query
68 | """
69 | # Ignore text after the SQL query terminator `;`
70 | assistant_message_content = assistant_message_content.split(";")[0]
71 |
72 | # Remove prefix for corrected query assistant message
73 | split_corrected_query_message = assistant_message_content.split(":")
74 | if len(split_corrected_query_message) > 1:
75 | sql_query = split_corrected_query_message[1].strip()
76 | else:
77 | sql_query = assistant_message_content
78 | return sql_query
79 |
80 |
81 | def extract_sql_query_from_message(assistant_message_content):
82 | print(assistant_message_content)
83 | content = extract_code_from_markdown(assistant_message_content)
84 | return clean_message_content(content)
85 |
86 |
87 | def extract_code_from_markdown(assistant_message_content):
88 | matches = re.findall(r"```([\s\S]+?)```", assistant_message_content)
89 |
90 | if matches:
91 | code_str = matches[0]
92 | match = re.search(r"(?i)sql\s+(.*)", code_str, re.DOTALL)
93 | if match:
94 | code_str = match.group(1)
95 | else:
96 | code_str = assistant_message_content
97 |
98 | return code_str
--------------------------------------------------------------------------------
/byod/api/app/visualization/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/caesarHQ/textSQL/7393e19b1458f1004d312713f5fe49968449e33c/byod/api/app/visualization/__init__.py
--------------------------------------------------------------------------------
/byod/api/app/visualization/routes.py:
--------------------------------------------------------------------------------
1 | from flask import Blueprint, jsonify, make_response, request
2 |
3 | from .utils import get_changed_vega, get_vega_lite_spec
4 |
5 | bp = Blueprint('visualization_bp', __name__)
6 |
7 |
8 | @bp.route("/viz", methods=["POST"])
9 | def get_visualization():
10 | """
11 | Get Vega-Lite spec from data
12 | """
13 | request_body = request.get_json()
14 | data = request_body.get("data")
15 |
16 | if not data:
17 | return make_response(jsonify({"error": "`data` is missing from request body"}), 400)
18 |
19 | vega_lite_spec = get_vega_lite_spec(data)
20 | return make_response(jsonify({"vega_lite_spec": vega_lite_spec}), 200)
21 |
22 |
23 | @bp.route('/text_to_viz', methods=['POST'])
24 | def modify_visualization():
25 | """
26 | Change Vega-Lite spec based on a command
27 | """
28 | request_body = request.get_json()
29 | natural_language_command = request_body.get('natural_language_command')
30 | vega_lite_spec = request_body.get('vega_lite_spec')
31 |
32 | if not natural_language_command:
33 | return make_response(jsonify({"error": "`natural_language_command` is missing from request body"}), 400)
34 |
35 | if not vega_lite_spec:
36 | return make_response(jsonify({"error": "`vega_lite_spec` is missing from request body"}), 400)
37 |
38 | changed_vega = get_changed_vega(natural_language_command, vega_lite_spec)
39 | return make_response(jsonify({"changed_vega": changed_vega}), 200)
--------------------------------------------------------------------------------
/byod/api/app/visualization/utils.py:
--------------------------------------------------------------------------------
1 | import json
2 | import re
3 | from typing import Dict
4 |
5 | from ..utils import get_assistant_message, get_few_shot_messages
6 |
7 |
8 | def make_default_visualize_data_messages():
9 | default_messages = [{
10 | "role": "system",
11 | "content": (
12 | "You are a helpful assistant for generating syntactically correct Vega-Lite specs that are best for visualizing given data."
13 | " Write responses in markdown format."
14 | " You will be given a JSON object in the following format."
15 | "\n\n"
16 | """
17 | {
18 | "fields": [
19 | {
20 | "name": "field_name", // name of the field
21 | "type": "nominal" // type of the field (quantitative, nominal, ordinal, temporal)
22 | "sample_value": "sample_value" // example value for the field
23 | }
24 | ],
25 | "total_rows": 100 // total number of rows in the result
26 | }
27 | """
28 | )
29 | }]
30 | default_messages.extend(get_few_shot_messages(mode="visualization"))
31 | return default_messages
32 |
33 |
34 | def make_visualize_data_message():
35 | return (
36 | "Generate a syntactically correct Vega-Lite spec to best visualize the given data."
37 | "\n\n"
38 | "{data}"
39 | )
40 |
41 |
42 | def make_default_visualization_change_messages():
43 | default_messages = [{
44 | "role": "system",
45 | "content": (
46 | "You are a helpful assistant for making changes to a Vega-Lite spec."
47 | " You generate a syntactically correct Vega-Lite spec."
48 | " You will be given a Vega-Lite spec and a command."
49 | " Write responses in markdown format."
50 | )
51 | }]
52 | default_messages.extend(get_few_shot_messages(mode="visualization_edits"))
53 | return default_messages
54 |
55 |
56 | def make_visualization_change_message():
57 | return (
58 | "Make the following changes to the given Vega-Lite spec to best visualize the data."
59 | "\n\n"
60 | "changes: {command}"
61 | "\n\n"
62 | "Vega-Lite spec: {vega_lite_spec}"
63 | )
64 |
65 |
66 | def get_vega_lite_spec(data) -> Dict:
67 | messages = make_default_visualize_data_messages()
68 | messages.append({
69 | "role": "user",
70 | "content": make_visualize_data_message().format(
71 | data=json.dumps(data, indent=2)
72 | )
73 | })
74 | vega = extract_json_str_from_markdown(
75 | get_assistant_message(messages)["message"]["content"]
76 | )
77 | return json.loads(vega)
78 |
79 |
80 | def get_changed_vega(command, vega_lite_spec) -> Dict:
81 | messages = make_default_visualization_change_messages()
82 | messages.append({
83 | "role": "user",
84 | "content": make_visualization_change_message().format(
85 | command=command,
86 | vega_lite_spec=vega_lite_spec
87 | )
88 | })
89 | vega = extract_json_str_from_markdown(
90 | get_assistant_message(messages)["message"]["content"]
91 | )
92 | return json.loads(vega)
93 |
94 |
95 | def extract_json_str_from_markdown(assistant_message_content) -> str:
96 | matches = re.findall(r"```([\s\S]+?)```", assistant_message_content)
97 |
98 | if matches:
99 | code_str = matches[0]
100 | match = re.search(r"(?i)bash\s+(.*)", code_str, re.DOTALL)
101 | if match:
102 | code_str = match.group(1)
103 | else:
104 | code_str = assistant_message_content
105 |
106 | return code_str
--------------------------------------------------------------------------------
/byod/api/requirements.txt:
--------------------------------------------------------------------------------
1 | Flask===2.2.2
2 | flask-admin==1.6.1
3 | flask-sqlalchemy==3.0.2
4 | flask-cors==3.0.10
5 | Flask-Migrate==4.0.4
6 | psycopg2-binary==2.9.5
7 | gunicorn==20.1.0
8 | openai[embeddings]==0.27.2
9 | openai==0.27.2
10 | python-dotenv==1.0.0
11 | blinker==1.5
12 | joblib==1.2.0
13 | requests==2.28.1
14 | pytest==7.2.2
15 | pinecone-client==2.2.1
16 | snowflake-connector-python==3.0.2
--------------------------------------------------------------------------------
/byod/api/scripts/dev.sh:
--------------------------------------------------------------------------------
1 | FLASK_APP=app.py FLASK_DEBUG=true flask run -p 9000
--------------------------------------------------------------------------------
/byod/api/scripts/setup.sh:
--------------------------------------------------------------------------------
1 | # create python venv named venv
2 | python3.10 -m venv ./venv
3 |
4 | # activate venv
5 | source venv/bin/activate
6 |
7 | # upgrade pip
8 | pip install --upgrade pip
9 |
10 | # install project dependencies
11 | cat requirements.txt | sed -e '/^\s*#.*$/d' -e '/^\s*$/d' | xargs -n 1 python -m pip install
12 |
--------------------------------------------------------------------------------
/byod/client/.env.example:
--------------------------------------------------------------------------------
1 | API_BASE="http://localhost:9000"
--------------------------------------------------------------------------------
/byod/client/.gitignore:
--------------------------------------------------------------------------------
1 | .env
2 | DS_Store
3 | venv/
4 | __pycache__/
5 | scratch/
--------------------------------------------------------------------------------
/byod/client/README.md:
--------------------------------------------------------------------------------
1 | # TextSQL front-end
2 |
3 | A front-end streamlit application for Text-to-SQL
4 |
5 | ## Prerequisites
6 | `python3.10`
7 |
8 | ## Required configuration for development:
9 | - base URL for TextSQL API
10 |
11 | Configure the above in `.env`. Refer to `.env.example`.
12 |
13 | ## Local development
14 |
15 | Initial setup
16 | ```
17 | $ ./scripts/setup.sh
18 | ```
19 |
20 | Activate virtual env
21 | ```
22 | $ source ./venv/bin/activate
23 | ```
24 |
25 | Run local instance
26 | ```
27 | $ ./scripts/dev.sh
28 | ```
--------------------------------------------------------------------------------
/byod/client/app.py:
--------------------------------------------------------------------------------
1 | import time
2 |
3 | import requests
4 | import streamlit as st
5 | from config import API_BASE
6 |
7 | VEGA_LITE_TYPES_MAP = {
8 | "int": "quantitative",
9 | "float": "quantitative",
10 | "str": "nominal",
11 | "bool": "nominal",
12 | "date": "temporal",
13 | "time": "temporal",
14 | "datetime": "temporal",
15 | }
16 |
17 |
18 | def create_viz_data_dict(column_names, column_types, results):
19 | data = {
20 | "fields": [],
21 | "total_rows": len(results),
22 | }
23 | for i, column_name in enumerate(column_names):
24 | data["fields"].append({
25 | "name": column_name,
26 | "type": VEGA_LITE_TYPES_MAP.get(column_types[i], "nominal"),
27 | })
28 | for i in range(len(results)):
29 | # include 1 sample
30 | if i == 1:
31 | break
32 | r = results[i]
33 | for j, column_name in enumerate(column_names):
34 | data["fields"][j]["sample_value"] = r[column_name]
35 | return data
36 |
37 |
38 | def main():
39 | st.title("Text-to-SQL")
40 |
41 | natural_language_query = st.text_input(label="Ask anything...", label_visibility="hidden", placeholder="Ask anything...")
42 |
43 | if natural_language_query:
44 | with st.spinner(text="Generating SQL..."):
45 | start_time = time.time()
46 | response = requests.post(f"{API_BASE}/text_to_sql", json={"natural_language_query": natural_language_query})
47 | end_time = time.time()
48 | time_taken = end_time - start_time
49 | if response.status_code == 200:
50 | st.info(f"SQL generated in {time_taken:.2f} seconds")
51 | SQL = f"""```sql
52 | {response.json().get("sql_query")}
53 | """
54 | st.markdown(SQL)
55 |
56 | RESULT = response.json().get("result", {})
57 | st.table(RESULT.get("results", []))
58 |
59 | with st.spinner(text="Generating visualization..."):
60 | start_time = time.time()
61 | response = requests.post(f"{API_BASE}/viz",
62 | json={
63 | "data": create_viz_data_dict(
64 | RESULT.get("column_names", []),
65 | RESULT.get("column_types", []),
66 | RESULT.get("results", []),
67 | )
68 | }
69 | )
70 | end_time = time.time()
71 | time_taken = end_time - start_time
72 | if response.status_code == 200:
73 | st.info(f"Visualization generated in {time_taken:.2f} seconds")
74 | VEGA_LITE_SPEC = response.json().get("vega_lite_spec")
75 | st.vega_lite_chart(data=RESULT.get("results", []), spec=VEGA_LITE_SPEC)
76 | else:
77 | st.error(f"{response.status_code}: {response.reason}")
78 | st.info("Sorry, I couldn't generate a visualization. Please try again.")
79 | else:
80 | st.error(f"{response.status_code}: {response.reason}")
81 | st.info("Sorry, I couldn't answer your question/command. Please try again.")
82 |
83 |
84 | if __name__ == "__main__":
85 | main()
--------------------------------------------------------------------------------
/byod/client/config.py:
--------------------------------------------------------------------------------
1 | from os import getenv
2 |
3 | from dotenv import load_dotenv
4 |
5 |
6 | load_dotenv()
7 |
8 | API_BASE = getenv("API_BASE")
--------------------------------------------------------------------------------
/byod/client/requirements.txt:
--------------------------------------------------------------------------------
1 | streamlit==1.18.1
2 | requests==2.28.2
3 | python-dotenv==1.0.0
--------------------------------------------------------------------------------
/byod/client/scripts/dev.sh:
--------------------------------------------------------------------------------
1 | streamlit run app.py
--------------------------------------------------------------------------------
/byod/client/scripts/setup.sh:
--------------------------------------------------------------------------------
1 | # create python venv named venv
2 | python3.10 -m venv ./venv
3 |
4 | # activate venv
5 | source venv/bin/activate
6 |
7 | # upgrade pip version
8 | pip install --upgrade pip
9 |
10 | # install project dependencies
11 | cat requirements.txt | sed -e '/^\s*#.*$/d' -e '/^\s*$/d' | xargs -n 1 python -m pip install
12 |
--------------------------------------------------------------------------------
/client/censusGPT/.env.production:
--------------------------------------------------------------------------------
1 | # `REACT_APP` prefix is required to expose to client-side
2 | REACT_APP_VERCEL_ANALYTICS_ID=$VERCEL_ANALYTICS_ID
--------------------------------------------------------------------------------
/client/censusGPT/.eslintignore:
--------------------------------------------------------------------------------
1 | **/node_modules/*
2 | **/out/*
3 | **/.next/*
4 |
--------------------------------------------------------------------------------
/client/censusGPT/.eslintrc.json:
--------------------------------------------------------------------------------
1 | {
2 | "env": {
3 | "browser": true,
4 | "es6": true
5 | },
6 | "extends": [
7 | "eslint:recommended",
8 | "plugin:react/recommended"
9 | ],
10 | "parserOptions": {
11 | "ecmaFeatures": {
12 | "jsx": true
13 | },
14 | "ecmaVersion": 2020,
15 | "sourceType": "module"
16 | },
17 | "plugins": [
18 | "react",
19 | "import"
20 | ],
21 | "rules": {
22 | "react/prop-types": 0,
23 | "no-undef": "error",
24 | "no-unused-vars": "warn",
25 | "react/no-unescaped-entities": "off",
26 | "no-irregular-whitespace": "off",
27 | "react/no-unknown-property": "off"
28 | },
29 | "globals": {
30 | "process": "readonly",
31 | "React": "writable",
32 | "require": "readonly"
33 | }
34 | }
--------------------------------------------------------------------------------
/client/censusGPT/.gitignore:
--------------------------------------------------------------------------------
1 | # See https://help.github.com/articles/ignoring-files/ for more about ignoring files.
2 |
3 | # dependencies
4 | /node_modules
5 | /.pnp
6 | .pnp.js
7 |
8 | # testing
9 | /coverage
10 |
11 | # production
12 | /build
13 | /dist
14 |
15 | # misc
16 | .DS_Store
17 | .env.local
18 | .env.development.local
19 | .env.test.local
20 | .env.production.local
21 |
22 | npm-debug.log*
23 | yarn-debug.log*
24 | yarn-error.log*
25 |
26 | # Environment Variables
27 | .env
28 | .env.build
29 | .vercel
30 |
31 | .idea/
--------------------------------------------------------------------------------
/client/censusGPT/.prettierignore:
--------------------------------------------------------------------------------
1 | node_modules
2 | # Ignore artifacts:
3 | build
4 | coverag
--------------------------------------------------------------------------------
/client/censusGPT/.prettierrc.json:
--------------------------------------------------------------------------------
1 | {
2 | "trailingComma": "es5",
3 | "tabWidth": 4,
4 | "semi": false,
5 | "singleQuote": true
6 | }
--------------------------------------------------------------------------------
/client/censusGPT/README.md:
--------------------------------------------------------------------------------
1 | # textSQL Frontend
2 | ## Development
3 |
4 | In the project directory, you can run:
5 |
6 | ### `npm run dev`
7 |
8 | Runs the app with **API pointing to Development environment**. Open [http://localhost:3000](http://localhost:3000) to view in browser.
9 |
10 | ### `npm start`
11 |
12 | Runs the app in development mode with **API pointing to production**. Open [http://localhost:3000](http://localhost:3000) to view in browser.
13 | The page will reload when you make changes. You may also see any lint errors in the console.
14 |
15 | ### `npm test`
16 |
17 | Launches the test runner in the interactive watch mode. See the section about [running tests](https://facebook.github.io/create-react-app/docs/running-tests) for more information.
18 |
19 | ### `npm run build`
20 |
21 | Builds the app for production to the `build` folder.
22 |
23 | It correctly bundles React in production mode and optimizes the build for the best performance. The build is minified and the filenames include the hashes.
--------------------------------------------------------------------------------
/client/censusGPT/package.json:
--------------------------------------------------------------------------------
1 | {
2 | "private": true,
3 | "dependencies": {
4 | "@headlessui/react": "^1.7.12",
5 | "@heroicons/react": "^2.0.16",
6 | "@sentry/react": "^7.42.0",
7 | "@sentry/tracing": "^7.42.0",
8 | "@tailwindcss/forms": "^0.5.3",
9 | "@testing-library/jest-dom": "^5.16.5",
10 | "@testing-library/react": "^13.4.0",
11 | "@testing-library/user-event": "^14.4.3",
12 | "@turf/turf": "^6.5.0",
13 | "autoprefixer": "^10.4.13",
14 | "deck.gl": "^8.8.26",
15 | "mapbox-gl": "^2.13.0",
16 | "plotly.js": "^2.18.2",
17 | "postcss-cli": "^10.1.0",
18 | "posthog-js": "^1.50.4",
19 | "react": "^18.2.0",
20 | "react-dom": "^18.2.0",
21 | "react-github-btn": "^1.4.0",
22 | "react-hot-toast": "^2.4.0",
23 | "react-icons": "^4.8.0",
24 | "react-map-gl": "^7.0.21",
25 | "react-plotly.js": "^2.6.0",
26 | "react-router-dom": "^6.8.2",
27 | "react-scripts": "5.0.1",
28 | "react-syntax-highlighter": "^15.5.0",
29 | "tailwindcss": "^3.2.7",
30 | "turf": "^3.0.14",
31 | "use-debounce": "^9.0.3",
32 | "web-vitals": "^3.0.4",
33 | "worker-loader": "^3.0.8",
34 | "@turf/bbox": "^6.5.0"
35 | },
36 | "devDependencies": {
37 | "eslint": "^8.36.0",
38 | "eslint-plugin-only-warn": "^1.1.0",
39 | "eslint-plugin-react": "^7.32.2",
40 | "prettier": "^2.8.4"
41 | },
42 | "scripts": {
43 | "start": "react-scripts start",
44 | "build": "react-scripts build",
45 | "dev": "REACT_APP_HOST_ENV=dev react-scripts start",
46 | "lint": "eslint ./ --ext js,jsx,ts,tsx",
47 | "test": "react-scripts test",
48 | "eject": "react-scripts eject",
49 | "format": "prettier --check ./src",
50 | "format:fix": "prettier --write ./src",
51 | "lint": "eslint ./src",
52 | "lint:fix": "eslint --fix ./src"
53 | },
54 | "eslintConfig": {
55 | "extends": [
56 | "react-app",
57 | "react-app/jest"
58 | ]
59 | },
60 | "browserslist": {
61 | "production": [
62 | ">0.2%",
63 | "not dead",
64 | "not op_mini all"
65 | ],
66 | "development": [
67 | "last 1 chrome version",
68 | "last 1 firefox version",
69 | "last 1 safari version"
70 | ]
71 | }
72 | }
73 |
--------------------------------------------------------------------------------
/client/censusGPT/postcss.config.js:
--------------------------------------------------------------------------------
1 | module.exports = {
2 | plugins: [require('tailwindcss'), require('autoprefixer')],
3 | };
4 |
--------------------------------------------------------------------------------
/client/censusGPT/public/favicon.ico:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/caesarHQ/textSQL/7393e19b1458f1004d312713f5fe49968449e33c/client/censusGPT/public/favicon.ico
--------------------------------------------------------------------------------
/client/censusGPT/public/index.html:
--------------------------------------------------------------------------------
1 |
2 |
3 |
4 |
5 |
6 |
7 |
8 |
12 |
13 |
17 |
18 |
27 | 7 | Hey! The team behind CensusGPT is now working on Julius, your personal AI data analyst 8 |
9 | 13 | Check out Julius 14 | 15 |39 | {example.input_text} 40 |
41 |Try these:
42 |Basic
44 |{ 54 | capturePosthog( 55 | 'example_clicked', 56 | { natural_language_query: q } 57 | ) 58 | props.setQuery(q) 59 | props.handleClick(q) 60 | }} 61 | > 62 | 66 |
{q}
67 | 68 |Advanced
82 |{ 92 | capturePosthog( 93 | 'example_clicked', 94 | { natural_language_query: q } 95 | ) 96 | props.setQuery(q) 97 | props.handleClick(q) 98 | }} 99 | > 100 | 104 |
{q}
105 | 106 |Try one of these examples:
19 |CaesarHQ operates the censusGPT.com website, which provides a service for generating SQL queries on census data.
6 |We collect information that you provide when using our website, including the SQL queries that you generate and any feedback that you provide. We use this information to improve our website and provide you with a better user experience.
8 |Like many website operators, we collect information that your browser sends whenever you visit our website. This may include information such as your computer's Internet Protocol (IP) address, browser type, browser version, the pages of our website that you visit, the time and date of your visit, the time spent on those pages and other statistics.
10 |We use cookies to collect information about your preferences and to personalize your experience on our website. You can instruct your browser to refuse all cookies or to indicate when a cookie is being sent. However, if you do not accept cookies, you may not be able to use some portions of our website.
12 |We may use third party services, such as Google Analytics, to collect, monitor and analyze usage of our website.
14 |The security of your personal information is important to us, but please remember that no method of transmission over the Internet, or method of electronic storage, is 100% secure. While we strive to use commercially acceptable means to protect your personal information, we cannot guarantee its absolute security.
16 |CaesarHQ reserves the right to modify this privacy policy at any time without notice. Your continued use of our website following any such modification constitutes your agreement to be bound by the modified privacy policy.
18 |If you have any questions about this privacy policy, please contact us at team@caesarhq.com.
20 |This privacy policy is effective as of April 10, 2023.
21 |Welcome to censusGPT.com, a website owned and operated by CaesarHQ. By accessing or using our website, you agree to be bound by the following terms and conditions:
7 |Our website is designed to help you generate SQL queries on census data for informational and research purposes only. You agree to use our website in accordance with all applicable laws and regulations.
9 |Our website and its entire contents, features, and functionality (including but not limited to all information, software, text, displays, images, video, and audio, and the design, selection, and arrangement thereof), are owned by CaesarHQ or its licensors and are protected by United States and international copyright, trademark, patent, trade secret, and other intellectual property or proprietary rights laws.
11 |Our website and its contents are provided on an "as is" and "as available" basis without any warranties of any kind, either express or implied, including but not limited to warranties of merchantability, fitness for a particular purpose, or non-infringement. CaesarHQ does not warrant that our website will be uninterrupted or error-free, that defects will be corrected, or that our website or the server that makes it available are free of viruses or other harmful components.
13 |In no event shall CaesarHQ, its affiliates, licensors, or service providers be liable for any direct, indirect, incidental, special, consequential, or punitive damages arising from or related to your use of our website, including but not limited to any errors or omissions in any content, or any loss or damage of any kind incurred as a result of the use of any content (or product) posted, transmitted, or otherwise made available via the website.
15 |You agree to indemnify, defend, and hold harmless CaesarHQ, its affiliates, licensors, and service providers, and its and their respective officers, directors, employees, contractors, agents, licensors, suppliers, successors, and assigns from and against any claims, liabilities, damages, judgments, awards, losses, costs, expenses, or fees (including reasonable attorneys' fees) arising out of or relating to your violation of these terms of service or your use of our website.
17 |CaesarHQ reserves the right to modify these terms of service at any time without notice. Your continued use of our website following any such modification constitutes your agreement to be bound by the modified terms of service.
19 |These terms of service and any dispute or claim arising out of or related to your use of our website shall be governed by and construed in accordance with the laws of the State of California, without giving effect to any principles of conflicts of law.
21 |If you have any questions about these terms of service, please contact us at team@caesarhq.com.
23 |