2 |
3 | Show user account details
4 |
5 |
6 | Token: {{ user?.access_token || "null" }}
7 |
8 |
9 | User: {{ user?.username || "null" }}
10 |
11 |
12 |
13 |
14 |
15 |
16 |
31 |
32 |
34 |
--------------------------------------------------------------------------------
/client/start:
--------------------------------------------------------------------------------
1 | #!/bin/bash
2 |
3 | pnpm i
4 | pnpm dev
--------------------------------------------------------------------------------
/client/vite.config.js:
--------------------------------------------------------------------------------
1 | import { defineConfig } from 'vite'
2 | import vue from '@vitejs/plugin-vue'
3 | import svgLoader from 'vite-svg-loader'
4 | import { resolve } from 'path'
5 | // https://vitejs.dev/config/
6 | export default defineConfig({
7 | plugins: [vue(), svgLoader()],
8 | resolve: {
9 | alias: {
10 | '@': resolve('src'),
11 | }
12 | },
13 | server: {
14 | proxy: {
15 | "/api": {
16 | target: "http://127.0.0.1:8000",
17 | changeOrigin: true,
18 | rewrite: path => path.replace(RegExp("/api"), ''),
19 | }
20 | }
21 | }
22 | })
23 |
--------------------------------------------------------------------------------
/docs/framework.md:
--------------------------------------------------------------------------------
1 | ## Learn how to organize frameworks
2 |
3 | 1.
4 | i have a `chatdata` web project consisting of a `server` folder and `client` folder. In `server` folder, I build a api backend server in `python` code where the `fastapi` and `uvicorn` package are used. and in `client` folder, i bulid a frontend website by `node, npm, pnpm, html, vue, js`.
5 |
6 | Now the `server` of `chatdata` project is implemention by python async function , but it only single user access. when there are multiple user access, it will response same content for all user. How do you extend this project to one that responds independently to multiple users？
7 |
8 | 2. How to transform this project into a multi-user independent response content project based on the current framework (front-end: fastapi, uvicorn, etc.; back-end vue, etc., as described above)
9 |
10 |
--------------------------------------------------------------------------------
/docs/install.md:
--------------------------------------------------------------------------------
1 | # Install
2 |
3 | 下载本项目
4 | ```shell
5 | git clone git@github.com:hbchen121/ChatYourData.git
6 | ```
7 |
8 | ## 前端安装（Frontend Install）
9 |
10 | ```shell
11 | cd client
12 | ```
13 |
14 | 安装nodejs
15 | ```shell
16 | # Linux
17 | curl -fsSL https://deb.nodesource.com/setup_16.x | sudo -E bash -
18 | sudo apt-get install -y nodejs
19 |
20 | # MacOS
21 | brew install nodejs
22 | ```
23 |
24 | 安装相关包，需要再client目录下
25 | ```shell
26 | sudo npm install -g pnpm
27 | sudo npm install vuex
28 | sudo npm install vue-router
29 | sudo npm install axios
30 | ```
31 |
32 | ```shell
33 | sudo bash start
34 | ```
35 |
36 | ## 后端安装（Backend Install）
37 |
38 | 建议先安装conda/miniconda进行python包管理（参考[miniconda_install](miniconda_install.md))，同时创建Python3.9环境
39 | ```shell
40 | conda create -n chatdata python==3.9
41 |
42 | # activate
43 | conda activate chatdata
44 | ```
45 |
46 | ```shell
47 | cd serve
48 |
49 | pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple/
50 | ```
51 |
52 | 在`start`中修改自己的代理端口，比如我的端口是 8889，否则无法访问 OPENAI_API
53 | ```shell
54 | export HTTP_PROXY=http://127.0.0.1:8889
55 | export HTTP_PROXY=http://127.0.0.1:8889
56 | ```
57 |
58 | 启动脚本
59 | ```shell
60 | bash sudo start
61 | ```
62 |
63 | 访问端口 `http://0.0.0.0:5173` 或 `http://127.0.0.1:5173` 即可进入
64 |
65 |
--------------------------------------------------------------------------------
/docs/miniconda_install.md:
--------------------------------------------------------------------------------
1 | # Miniconda 安装 (BY ChatGPT)
2 |
3 | 安装Miniconda是在Ubuntu上设置Python环境的一种常见方法。以下是在Ubuntu上安装Miniconda的简单教程：
4 |
5 | 1.下载Miniconda安装脚本：
6 | 前往Miniconda[官方网站](https://docs.conda.io/en/latest/miniconda.html) 下载适用于Linux的Miniconda安装脚本。根据您的系统架构选择合适的版本（通常为64位）。
7 |
8 | 2.打开终端：
9 | 在Ubuntu上打开终端应用程序。您可以通过按下Ctrl + Alt + T键盘快捷键来快速启动终端。
10 |
11 | 3.运行安装脚本：
12 | 使用以下命令在终端中运行Miniconda安装脚本。请注意，根据您下载的文件名，可能需要调整命令中的文件名。
13 |
14 | ```shell
15 | bash Miniconda3-latest-Linux-x86_64.sh
16 | ```
17 | 按照提示进行安装：
18 | 运行安装脚本后，您将看到一系列安装提示。按照屏幕上的指示，可以通过按Enter键进行默认安装。您可以选择接受许可协议并选择安装路径。通常，使用默认选项即可。
19 |
20 | 4.初始化Conda：
21 | 安装完成后，您需要初始化Conda环境。按照终端上的提示，运行以下命令：
22 |
23 | ```shell
24 | source ~/.bashrc
25 | ```
26 | 验证安装：
27 | 运行以下命令来验证Miniconda的安装是否成功：
28 |
29 | ```shell
30 | conda --version
31 | ```
32 |
33 | 如果安装成功，将显示Conda的版本号。
34 |
35 | 现在，您已成功在Ubuntu上安装了Miniconda。您可以使用Conda命令来创建和管理Python环境，安装软件包以及执行其他与环境相关的任务。有关更多详细信息和使用示例，请参阅Miniconda官方文档（https://docs.conda.io/en/latest/miniconda.html）。
36 |
37 |
38 |
39 |
40 |
41 |
--------------------------------------------------------------------------------
/docs/todo.md:
--------------------------------------------------------------------------------
1 | ## DONE and TODO
2 |
3 | ### DONE
4 | 1. 单用户访问
5 | 2. OPENAI API KEY 外放，方便用户使用
6 | 3. 简单的文件助手交互访问
7 |
8 | ### TODO
9 | 1. 添加用户验证功能，实现多用户并行交互
10 | 1. 前端添加用户验证UI（cookies? token?）
11 | 2. 后端添加不同用户分别映射功能
12 | 2. 添加不同模式下的 DataChat
13 | 1. 后端添加不同场景下的交互接口
14 | 2. 前端添加相应的接口，暴露用户的需求
15 | 3.
--------------------------------------------------------------------------------
/login.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/hbchen121/ChatYourData/8a378f2966d28eb160f6ccd35d0bcbd8a8934378/login.png
--------------------------------------------------------------------------------
/overview.jpeg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/hbchen121/ChatYourData/8a378f2966d28eb160f6ccd35d0bcbd8a8934378/overview.jpeg
--------------------------------------------------------------------------------
/preview.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/hbchen121/ChatYourData/8a378f2966d28eb160f6ccd35d0bcbd8a8934378/preview.png
--------------------------------------------------------------------------------
/readme.md:
--------------------------------------------------------------------------------
1 | # ChatYourData
2 |
3 | 
4 |
5 | ChatYourData ([github](https://github.com/hbchen121/ChatYourData)) 是一个基于ChatGPT等大模型技术的数据处理网站，通过上传用户私有数据、建立索引，用户可基于私有数据与ChatGPT等大模型进行对话，使大模型具有私有数据知识先验的前提下对用户的问题进行解答。具体的原理可参考：
6 | - [如何让 ChatGPT(LLMs) 学习更多的私有数据知识？（一）](https://mp.weixin.qq.com/s/rpQMZmAeFzNFEYMBQgp_Ng)
7 | - [如何让 ChatGPT(LLMs) 学习更多的私有数据知识？（二）](https://mp.weixin.qq.com/s/2iu8x5RlcWTLu9yCR4ChRg)
8 |
9 | ## ChatYourData 特点
10 |
11 | - 网页部署地址：[link](http://101.42.32.28:5173/)（域名待续中）
12 | - 通过 ChatYourData 可以使 ChatGPT 学习到用户私有知识；
13 | - ChatYourData 需要用户使用自己的 OPEN API KEY，在无须魔法的情况下，也能与 ChatGPT 对话；
14 |
15 | ## 登录
16 | - 输入用户名和密码即可自动注册登录，用户名和密码无限制，主要是为了保护个人的配置和文件；
17 |
18 | 
19 |
20 | ## 用法
21 | 1. 登录完成后首先需要设置自己的 OPENAI API KEY，否则无法进行搜索，API KEY 获取可参考教程。设置 API KEY 后可以刷新一下页面，就能看到自己的 API KEY显示出来；
22 | 2. 将需要搜索（Chat）的数据（文件或链接）进行上传/添加：
23 | - 文件目前支持 .pdf, .epub, .md, .txt 类型文件
24 | - 链接对目标网站有要求，有些网站不能正常获取（比如github，csdn等）
25 | 3. 添加后，会出现一个文件项，同时index为 doing 状态，可以刷新下查看 index 状态变化。当状态为done时即可基于索引与数据内容进行交流。
26 |
27 | 
28 |
29 | ## 构建
30 |
31 | 构建自己的 ChatYourData 服务见[安装教程](./docs/install.md)，你需要：
32 | - 自己的代理节点（不然无法访问OPENAI API）
33 |
34 | 基于本项目进行二次开发，你需要：
35 | - 前端：Vue + JavaScript + Html
36 | - 后端：Python + SQL + AI（可选，调接口就不需要，优化索引逻辑等可能需要，非必须）
37 |
38 |
39 | ## 合作
40 |
41 | - 本项目的前端、后端交互等主要靠 ChatGPT 和 Claude 辅助编程来debug，我是不咋会前端...有相关AI辅助编程需要的朋友也欢迎交流..
42 | - 欢迎有对这个项目感兴趣的志同道合的朋友交流，目前网站还有许多需要优化的地方，期待大家批评交流~
43 |
44 | ## 关于我
45 | 另外，介绍下自己，我是卷了又没卷，薛定谔的卷的AI算法工程师「陈城南」~
46 | 担任某大厂的算法工程师，带来最新的前沿AI知识和工具，欢迎大家交流~，后续我还会分享更多 AI 有趣工具和实用玩法，包括 ChatGPT、AI绘图等。
47 | - 公众号「陈城南」
48 | - 加V「cchengnan113」备注 AI交流群 进行交流
49 | - 知乎账号「[陈城南](https://www.zhihu.com/people/cchengnan113)」
--------------------------------------------------------------------------------
/server/chatdata/__init__.py:
--------------------------------------------------------------------------------
1 | # encoding:utf-8
2 |
--------------------------------------------------------------------------------
/server/chatdata/chat_test.py:
--------------------------------------------------------------------------------
1 | # encoding:utf-8
2 | import os
3 | import os.path as osp
4 | import time
5 |
6 | from llama_index import GPTSimpleVectorIndex, SimpleDirectoryReader
7 | from langchain import OpenAI
8 | import logging
9 | import sys
10 |
11 |
12 | logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)
13 | logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))
14 |
15 |
16 | def chat_data(input_text, filename):
17 | time.sleep(3)
18 | return input_text
19 | # documents = SimpleDirectoryReader(input_files=[filename]).load_data()
20 | # os.environ['OPENAI_API_KEY'] = 'OPENAI_API_KEY'
21 | # index_file = f"cache/{osp.basename(filename)}.json"
22 | # if osp.exists(index_file):
23 | # index = GPTSimpleVectorIndex.load_from_disk(index_file)
24 | # else:
25 | # index = GPTSimpleVectorIndex.from_documents(documents)
26 | # index.save_to_disk(index_file, encoding='utf-8')
27 | # response = index.query(input_text)
28 | # return response
29 |
30 |
31 | if __name__ == '__main__':
32 | import IPython
33 | filename = 'cache/未命名文本.txt'
34 | input_text = '文章标题是什么？'
35 | rsp = chat_data(input_text, filename)
36 | print(rsp.response)
37 | IPython.embed()
38 | pass
39 |
--------------------------------------------------------------------------------
/server/database/__init__.py:
--------------------------------------------------------------------------------
1 | # encoding:utf-8
2 |
--------------------------------------------------------------------------------
/server/database/database.db:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/hbchen121/ChatYourData/8a378f2966d28eb160f6ccd35d0bcbd8a8934378/server/database/database.db
--------------------------------------------------------------------------------
/server/database/db_docs.py:
--------------------------------------------------------------------------------
1 | import sqlite3
2 | import calendar
3 | import time
4 |
5 |
6 | class Docs:
7 |
8 | def __init__(self, username: str, uid: int, doc_id: str, doc_name: str, doc_type: str, size: int = 0,
9 | state: int = 0, create_at: int = 0, update_at: int = 0, id: int = 0, userid: int = 0):
10 | self.uid = uid
11 | self.doc_id = doc_id
12 | self.doc_name = doc_name
13 | self.doc_type = doc_type
14 | self.state = state
15 | self.size = size
16 | self.create_at = create_at
17 | self.update_at = update_at
18 | self.userid = userid
19 | self.username = username
20 |
21 | ts = calendar.timegm(time.gmtime())
22 | if self.create_at == 0:
23 | self.create_at = ts
24 | if self.update_at == 0:
25 | self.update_at = ts
26 |
27 | self.__conn = sqlite3.connect('database.db')
28 | self.__cursor = self.__conn.cursor()
29 |
30 | # 将对象插入到数据库中
31 | def insert(self):
32 | docs = self.get_by_doc_id(self.doc_id)
33 | if docs != None:
34 | return
35 | print('size', self.size)
36 | self.__cursor.execute(
37 | 'INSERT INTO docs (uid, doc_id, doc_name, doc_type, size, state, create_at, update_at, userid, username) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)',
38 | (self.uid, self.doc_id, self.doc_name, self.doc_type, self.size, self.state, self.create_at, self.update_at,
39 | self.userid, self.username))
40 | self.__conn.commit()
41 |
42 | # 从数据库中删除对象
43 | def delete(self):
44 | self.__cursor.execute('DELETE FROM docs WHERE doc_id = ?', (self.doc_id,))
45 | self.__conn.commit()
46 |
47 | # 更新对象在数据库中的信息
48 | def update(self):
49 | self.__cursor.execute(
50 | 'UPDATE docs SET uid = ?, doc_name = ?, doc_type = ?, size = ?, state = ?, create_at = ?, update_at = ?, userid = ?, username = ? WHERE doc_id = ?',
51 | (self.uid, self.doc_name, self.doc_type, self.size, self.state, self.create_at, self.update_at, self.userid,
52 | self.username, self.doc_id))
53 | self.__conn.commit()
54 |
55 | # 从数据库中获取所有对象
56 | @staticmethod
57 | def get_all():
58 | conn = sqlite3.connect('database.db')
59 | cur = conn.cursor()
60 | cur.execute('SELECT id,uid,doc_id,doc_name,doc_type,size,state,userid,username FROM docs')
61 | rows = cur.fetchall()
62 | cur.close()
63 | conn.close()
64 | return [{"id": row[0], "uid": row[1], "doc_id": row[2], "doc_name": row[3], "doc_type": row[4],
65 | "size": row[5], "state": row[6], "userid": row[7], "username": row[8]} for row in rows]
66 |
67 | @staticmethod
68 | def get_username_all(username):
69 | """
70 | 取出指定user的所有
71 | """
72 | conn = sqlite3.connect('database.db')
73 | cur = conn.cursor()
74 | cur.execute('SELECT id,uid,doc_id,doc_name,doc_type,size,state,userid,username FROM docs WHERE username = ?',
75 | (username,))
76 | rows = cur.fetchall()
77 | cur.close()
78 | conn.close()
79 | return [{"id": row[0], "uid": row[1], "doc_id": row[2], "doc_name": row[3], "doc_type": row[4],
80 | "size": row[5], "state": row[6], "userid": row[7], "username": row[8]} for row in rows]
81 |
82 | # 从数据库中获取指定doc_id的对象
83 | @staticmethod
84 | def get_by_doc_id(doc_id):
85 | conn = sqlite3.connect('database.db')
86 | cur = conn.cursor()
87 | cur.execute('SELECT id,uid,doc_id,doc_name,doc_type,size,state,userid,username FROM docs WHERE doc_id = ?',
88 | (doc_id,))
89 | row = cur.fetchone()
90 | cur.close()
91 | conn.close()
92 | if row is not None:
93 | return {"id": row[0], "uid": row[1], "doc_id": row[2], "doc_name": row[3], "doc_type": row[4],
94 | "size": row[5], "state": row[6], "userid": row[7], "username": row[8]}
95 | else:
96 | return None
97 |
98 | @staticmethod
99 | def del_by_doc_id(doc_id):
100 | conn = sqlite3.connect('database.db')
101 | cur = conn.cursor()
102 | cur.execute('DELETE FROM docs WHERE doc_id = ?', (doc_id,))
103 | cur.close()
104 | conn.commit()
105 | conn.close()
106 |
107 | def __str__(self):
108 | return f"Doc(uid={self.uid}, doc_id='{self.doc_id}', doc_name='{self.doc_name}', doc_type='{self.doc_type}', state={self.state}, create_at={self.create_at}, update_at={self.update_at})"
109 |
--------------------------------------------------------------------------------
/server/database/db_init.py:
--------------------------------------------------------------------------------
1 | import sqlite3
2 | import os
3 |
4 | os.remove("database.db")
5 |
6 | table_docs = """
7 | create table if not exists docs
8 | (
9 | id integer primary key autoincrement,
10 | uid integer default 0 not null,
11 | doc_id text default '' not null unique,
12 | doc_name text default '' not null,
13 | doc_type text default '' not null,
14 | state integer default 0 not null, -- 1 解析文本成功, 2 构建所有成功
15 | size integer default 0 not null,
16 | create_at integer default 0 not null,
17 | update_at integer default 0 not null,
18 | userid integer default 0 not null,
19 | username text default '' not null
20 | );
21 | """
22 |
23 | table_msg = """
24 | create table if not exists messages
25 | (
26 | id integer primary key autoincrement,
27 | uid integer default 0 not null,
28 | doc_id text default '' not null,
29 | role text default '' not null,
30 | content text default '' not null,
31 | create_at integer not null
32 | );
33 | """
34 |
35 | table_user = """
36 | create table if not exists users
37 | (
38 | id integer primary key autoincrement,
39 | userid integer default 0 not null,
40 | username text default '' not null,
41 | password text default '' not null,
42 | token text default '' not null,
43 | api_key text default '' not null,
44 | create_at integer not null
45 | );
46 | """
47 |
48 | con = sqlite3.connect("database.db")
49 |
50 | cur = con.cursor()
51 |
52 | cur.execute(table_docs)
53 | cur.execute(table_msg)
54 | cur.execute(table_user)
55 |
56 | res = cur.execute("SELECT name FROM sqlite_master")
57 |
58 | result = res.fetchall()
59 |
60 | print(result)
--------------------------------------------------------------------------------
/server/database/db_msg.py:
--------------------------------------------------------------------------------
1 | import sqlite3
2 | import calendar
3 | import time
4 | import json
5 |
6 | class Msg:
7 |
8 | def __init__(self, uid: int, doc_id: str, role: str, content, create_at: int = 0):
9 | self.uid = uid
10 | self.doc_id = doc_id
11 | self.role = role
12 | if role == 'chatdata':
13 | self.content = content.response
14 | else:
15 | self.content = content
16 | self.create_at = create_at
17 |
18 | ts = calendar.timegm(time.gmtime())
19 | if self.create_at == 0:
20 | self.create_at = ts
21 |
22 | self.__conn = sqlite3.connect('database.db')
23 | self.__cursor = self.__conn.cursor()
24 |
25 | # 将对象插入到数据库中
26 | def insert(self):
27 | self.__cursor.execute('INSERT INTO messages (uid, doc_id, role, content, create_at) VALUES (?, ?, ?, ?, ?)',
28 | (self.uid, self.doc_id, self.role, self.content, self.create_at))
29 | self.__conn.commit()
30 |
31 | # 从数据库中删除对象
32 | def delete(self):
33 | self.__cursor.execute(
34 | 'DELETE FROM messages WHERE doc_id = ?', (self.doc_id,))
35 | self.__conn.commit()
36 |
37 | # 从数据库中获取指定doc_id的对象
38 | @staticmethod
39 | def get_by_doc_id(doc_id, uid):
40 | conn = sqlite3.connect('database.db')
41 | cur = conn.cursor()
42 | cur.execute(
43 | 'SELECT uid, doc_id, role, content, create_at FROM messages WHERE doc_id = ? and uid = ?', (doc_id, uid))
44 | rows = cur.fetchall()
45 | cur.close()
46 | conn.close()
47 | if rows is not None:
48 | return [{"uid": row[0], "doc_id": row[1], "role": row[2], "content": row[3] if row[2] == 'user' else row[3]} for row in rows]
49 | else:
50 | return None
51 |
--------------------------------------------------------------------------------
/server/database/db_user.py:
--------------------------------------------------------------------------------
1 | import sqlite3
2 | import calendar
3 | import time
4 | from utils.passwords import hash
5 |
6 |
7 | class Users:
8 | def __init__(self, userid: int, username: str, password: str = '', token: str = '', api_key: str = '',
9 | create_at: int = 0, id: int = 0):
10 | self.userid = userid
11 | self.username = username
12 | self.password = password
13 | self.token = token
14 | self.api_key = api_key
15 | self.create_at = create_at
16 |
17 | ts = calendar.timegm(time.gmtime())
18 | if self.create_at == 0:
19 | self.create_at = ts
20 |
21 | self.__conn = sqlite3.connect('database.db')
22 | self.__cursor = self.__conn.cursor()
23 |
24 | # 将对象插入到数据库中
25 | def create(self):
26 | docs = self.get_by_username(self.username)
27 | if docs is not None:
28 | return self
29 | print('create username', self.username)
30 | if self.password == '':
31 | print("password is empty")
32 | return None
33 | self.password = hash(self.password)
34 | self.__cursor.execute(
35 | 'INSERT INTO users (userid,username,password,token,api_key, create_at) VALUES (?, ?, ?, ?, ?, ?)',
36 | (self.userid, self.username, self.password, self.token, self.api_key, self.create_at))
37 | self.__conn.commit()
38 | return self
39 |
40 | # 从数据库中删除对象
41 | def delete(self):
42 | self.__cursor.execute('DELETE FROM users WHERE username = ?', (self.username,))
43 | self.__conn.commit()
44 |
45 | #
46 | def update(self, password='', token='', api_key=''):
47 | """
48 | 更新对象在数据库中的信息，只能更新这三个
49 | """
50 | db_info = self.get_by_username(self.username)
51 | if db_info is None:
52 | print(f"get {self.username} info error")
53 | return
54 | self.password = password or db_info["password"]
55 | self.token = token or db_info["token"]
56 | self.api_key = api_key or db_info["api_key"]
57 | self.__cursor.execute('UPDATE users SET password = ?, token = ?, api_key = ? WHERE username = ?',
58 | (self.password, self.token, self.api_key, self.username))
59 | self.__conn.commit()
60 | print(f"update {self.username} success")
61 |
62 | # 从数据库中获取所有对象
63 | @staticmethod
64 | def get_all():
65 | conn = sqlite3.connect('database.db')
66 | cur = conn.cursor()
67 | cur.execute('SELECT id,userid,username,password,token,api_key FROM users')
68 | rows = cur.fetchall()
69 | cur.close()
70 | conn.close()
71 | return [
72 | {"id": row[0], "userid": row[1], "username": row[2], "password": row[3], "token": row[4], "api_key": row[5]}
73 | for row in rows]
74 |
75 | # 从数据库中获取指定doc_id的对象
76 | @staticmethod
77 | def get_by_username(username):
78 | conn = sqlite3.connect('database.db')
79 | cur = conn.cursor()
80 | cur.execute('SELECT id,userid,username,password,token,api_key FROM users WHERE username = ?', (username,))
81 | row = cur.fetchone()
82 | cur.close()
83 | conn.close()
84 | if row is not None:
85 | return {"id": row[0], "userid": row[1], "username": row[2], "password": row[3], "token": row[4],
86 | "api_key": row[5]}
87 | else:
88 | return None
89 |
90 | @staticmethod
91 | def del_by_username(username):
92 | conn = sqlite3.connect('database.db')
93 | cur = conn.cursor()
94 | cur.execute('DELETE FROM users WHERE username = ?', (username,))
95 | cur.close()
96 | conn.commit()
97 | conn.close()
98 |
99 | def __str__(self):
100 | return f"User(userid={self.userid}, username='{self.username}', password='{self.password}', token='{self.token}', api_key={self.api_key}, create_at={self.create_at}"
101 |
102 |
103 | def get_by_token(token):
104 | conn = sqlite3.connect('database.db')
105 | cur = conn.cursor()
106 | cur.execute('SELECT id,userid,username,password,token,api_key,create_at FROM users WHERE token = ?', (token,))
107 | row = cur.fetchone()
108 | cur.close()
109 | conn.close()
110 | if row is not None:
111 | return {"id": row[0], "userid": row[1], "username": row[2], "password": row[3], "token": row[4],
112 | "api_key": row[5], "create_at": row[6]}
113 | else:
114 | return None
115 |
116 |
117 | def token2user(token):
118 | info = get_by_token(token)
119 | if info is None:
120 | return info
121 | return Users(**info)
122 |
123 |
124 | if __name__ == '__main__':
125 | a = Users(0, "test", "test")
126 | a.create()
127 | users = Users.get_all()
128 | print(users)
129 |
--------------------------------------------------------------------------------
/server/download_website.py:
--------------------------------------------------------------------------------
1 | import os
2 | import shutil
3 | import requests
4 | from bs4 import BeautifulSoup
5 |
6 | url = "https://www.jianshu.com/p/474f4713753ct"
7 | response = requests.get(url)
8 |
9 | # 创建目录
10 | path = "example"
11 | if not os.path.exists(path):
12 | os.mkdir(path)
13 |
14 | # 解析 HTML 页面
15 | soup = BeautifulSoup(response.text, "html.parser")
16 |
17 | # 保存 HTML 页面
18 | with open(f"{path}/index.html", "w") as f:
19 | f.write(soup.prettify())
20 |
21 | # 保存 CSS 文件
22 | for link in soup.find_all("link"):
23 | if link.get("rel") == ["stylesheet"]:
24 | css_url = link.get("href")
25 | css_response = requests.get(css_url)
26 | css_path = f"{path}/{css_url.split('/')[-1]}"
27 | with open(css_path, "wb") as f:
28 | f.write(css_response.content)
29 |
30 | # 保存 JavaScript 文件
31 | for script in soup.find_all("script"):
32 | if script.get("src"):
33 | js_url = script.get("src")
34 | js_response = requests.get(js_url)
35 | js_path = f"{path}/{js_url.split('/')[-1]}"
36 | with open(js_path, "wb") as f:
37 | f.write(js_response.content)
38 |
39 | # 复制所有图片文件
40 | for img in soup.find_all("img"):
41 | img_url = img.get("src")
42 | img_response = requests.get(img_url)
43 | img_path = f"{path}/{img_url.split('/')[-1]}"
44 | with open(img_path, "wb") as f:
45 | f.write(img_response.content)
46 |
47 | # 复制所有媒体文件
48 | for source in soup.find_all("source"):
49 | media_url = source.get("src")
50 | media_response = requests.get(media_url)
51 | media_path = f"{path}/{media_url.split('/')[-1]}"
52 | with open(media_path, "wb") as f:
53 | f.write(media_response.content)
54 |
55 | # 复制所有字体文件
56 | for font in soup.find_all("link"):
57 | if font.get("rel") == ["stylesheet"] and "font" in font.get("href"):
58 | font_url = font.get("href")
59 | font_response = requests.get(font_url)
60 | font_path = f"{path}/{font_url.split('/')[-1]}"
61 | with open(font_path, "wb") as f:
62 | f.write(font_response.content)
63 |
64 | # 复制所有其他文件
65 | for link in soup.find_all("link"):
66 | if not link.get("rel") == ["stylesheet"]:
67 | file_url = link.get("href")
68 | if file_url and "." in file_url:
69 | file_extension = file_url.split(".")[-1]
70 | if file_extension not in ["css", "js"]:
71 | file_response = requests.get(file_url)
72 | file_path = f"{path}/{file_url.split('/')[-1]}"
73 | with open(file_path, "wb") as f:
74 | f.write(file_response.content)
75 |
--------------------------------------------------------------------------------
/server/models.py:
--------------------------------------------------------------------------------
1 | from typing import Optional
2 | from pydantic import BaseModel
3 | from sqlalchemy import (Column, PrimaryKeyConstraint, String)
4 | from sqlalchemy.ext.declarative import declarative_base
5 |
6 | Base = declarative_base()
7 |
8 | # User Accounts
9 |
10 | class UserModel(BaseModel):
11 | username: str
12 |
13 | class Config:
14 | orm_mode = True
15 |
16 | class UserWithHashModel(UserModel):
17 | password: str
18 |
19 | class UserOrm(Base):
20 | __tablename__ = 'users'
21 | username = Column(String, unique=True)
22 | password = Column(String)
23 | PrimaryKeyConstraint(username)
24 |
25 | def __repr__(self):
26 | return "