6 |
7 |
8 | ## Getting started
9 |
10 | Create a slack bot involves 3 steps
11 |
12 | * Create slack user
13 | * Set environment variables
14 | * Run the app locally
15 |
16 | ### Step 1: Create slack user token
17 |
18 | Follow the steps given below to fetch your slack user token to get data through Slack APIs:
19 |
20 | 1. Create a workspace on Slack if you don’t have one already by clicking [here](https://slack.com/intl/en-in/).
21 | 2. Create a new App on your Slack account by going [here](https://api.slack.com/apps).
22 | 3. Select `From Scratch`, then enter the App Name and select your workspace.
23 | 4. Navigate to `OAuth & Permissions` tab from the left sidebar and go to the `scopes` section. Add the following scopes under `User Token Scopes`:
24 |
25 | ```
26 | # Following scopes are needed for reading channel history
27 | channels:history
28 | channels:read
29 |
30 | # Following scopes are needed to fetch list of channels from slack
31 | groups:read
32 | mpim:read
33 | im:read
34 | ```
35 |
36 | 5. Click on the `Install to Workspace` button under `OAuth Tokens for Your Workspace` section in the same page and install the app in your slack workspace.
37 | 6. After installing the app you will see the `User OAuth Token`, save that token as you will need to configure it as `SLACK_USER_TOKEN` for this demo.
38 |
39 | ### Step 2: Set environment variables
40 |
41 | Navigate to `api` folder and set your `HUGGINGFACE_ACCESS_TOKEN` and `SLACK_USER_TOKEN` in `.env.example` file. Then rename the `.env.example` file to `.env`.
42 |
43 |
44 | ### Step 3: Run app locally
45 |
46 | Follow the instructions given below to run app locally based on your development setup (with docker or without docker):
47 |
48 | #### With docker
49 |
50 | ```bash
51 | docker-compose build
52 | ec start --docker
53 | ```
54 |
55 | #### Without docker
56 |
57 | ```bash
58 | ec install-reqs
59 | ec start
60 | ```
61 |
62 | Finally, you will have the Slack AI frontend running on http://localhost:3000. You can also access the REST APIs on http://localhost:8000.
63 |
64 | ## Credits
65 |
66 | This demo was built using the Embedchain's [full stack demo template](https://docs.embedchain.ai/get-started/full-stack). Follow the instructions [given here](https://docs.embedchain.ai/get-started/full-stack) to create your own full stack RAG application.
67 |
--------------------------------------------------------------------------------
/slack-ai/api/.env.example:
--------------------------------------------------------------------------------
1 | SLACK_USER_TOKEN=
2 | # set your openai token if you want to use openai LLMs
3 | # OPENAI_API_KEY=
4 | # Set your HuggingFace access token if you want to use open source LLMs
5 | HUGGINGFACE_ACCESS_TOKEN=
6 |
--------------------------------------------------------------------------------
/slack-ai/api/Dockerfile:
--------------------------------------------------------------------------------
1 | FROM python:3.11-slim
2 |
3 | WORKDIR /app
4 |
5 | COPY requirements.txt /app/
6 |
7 | RUN pip install -r requirements.txt
8 |
9 | COPY . /app
10 |
11 | EXPOSE 8000
12 |
13 | CMD ["python", "-m", "main"]
14 |
--------------------------------------------------------------------------------
/slack-ai/api/main.py:
--------------------------------------------------------------------------------
1 | import uvicorn
2 | from dotenv import load_dotenv
3 | from fastapi import FastAPI
4 |
5 | from routes import admin, api
6 |
7 | load_dotenv(".env")
8 |
9 | app = FastAPI(title="Embedchain API")
10 |
11 | app.include_router(api.router)
12 | app.include_router(admin.router)
13 |
14 | if __name__ == "__main__":
15 | uvicorn.run("main:app", host="0.0.0.0", port=8000, log_level="info",
16 | reload=True, timeout_keep_alive=600)
17 |
--------------------------------------------------------------------------------
/slack-ai/api/requirements.txt:
--------------------------------------------------------------------------------
1 | fastapi==0.108.0
2 | uvicorn==0.25.0
3 | embedchain[slack]
4 | beautifulsoup4
5 | sentence-transformers
6 |
--------------------------------------------------------------------------------
/slack-ai/api/routes/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/mem0ai/examples/f33f9c347abdab3a18d833f8392cbb4f7643d2ab/slack-ai/api/routes/__init__.py
--------------------------------------------------------------------------------
/slack-ai/api/routes/admin.py:
--------------------------------------------------------------------------------
1 | import chromadb
2 | from chromadb.config import Settings
3 | from fastapi import APIRouter
4 |
5 | router = APIRouter()
6 |
7 |
8 | chroma_settings = Settings(
9 | anonymized_telemetry=False,
10 | persist_directory="db",
11 | allow_reset=False,
12 | is_persistent=True,
13 | )
14 | client = chromadb.Client(chroma_settings)
15 |
16 |
17 | @router.get("/api/v1/admin/collections")
18 | async def get_all_collections():
19 | # Currently only works for ChromaDB but can be extended easily
20 | # for other vector stores as well
21 | collections = client.list_collections()
22 | responses = [c.dict() for c in collections]
23 | return responses
24 |
25 |
26 | # TODO(deshraj): Add pagination and make this endpoint agnostic to the vector store
27 | @router.get("/api/v1/admin/collections/chromadb/{collection_name}")
28 | async def get_collection_details(collection_name: str):
29 | collection = client.get_collection(collection_name)
30 | collection_data = collection.get()
31 | metadatas, documents = collection_data['metadatas'], collection_data['documents']
32 | collated_data = []
33 | for i in zip(metadatas, documents):
34 | collated_data.append({
35 | "metadata": i[0],
36 | "document": i[1]
37 | })
38 | response = {"details": collection.dict(), "data": collated_data}
39 | return response
40 |
--------------------------------------------------------------------------------
/slack-ai/api/routes/api.py:
--------------------------------------------------------------------------------
1 | import os
2 |
3 | from fastapi import APIRouter, Query, responses
4 | from pydantic import BaseModel
5 |
6 | from utils.app import chunker, ec_app, loader
7 |
8 | router = APIRouter()
9 |
10 |
11 | class SourceModel(BaseModel):
12 | channel: str
13 |
14 |
15 | class QuestionModel(BaseModel):
16 | question: str
17 | session_id: str
18 |
19 |
20 | @router.post("/api/v1/add")
21 | async def add_channel_data(source_model: SourceModel):
22 | """
23 | Adds a new source to the Embedchain app.
24 | Expects a JSON with a "channel" key.
25 | """
26 | channel = source_model.channel.replace("#", "").strip()
27 | try:
28 | ec_app.add(f"in:{channel}", data_type="slack", loader=loader, chunker=chunker)
29 | return {"message": f"Data for '{channel}' added successfully."}
30 | except Exception as e:
31 | response = f"An error occurred: Error message: {str(e)}. Contact Embedchain founders on Slack: https://embedchain.com/slack or Discord: https://embedchain.com/discord" # noqa:E501
32 | return {"message": response}
33 |
34 |
35 | @router.get("/api/v1/chat")
36 | async def handle_chat(query: str, session_id: str = Query(None)):
37 | """
38 | Handles a chat request to the Embedchain app.
39 | Accepts 'query' and 'session_id' as query parameters.
40 | """
41 | try:
42 | answer, metadata = ec_app.chat(query, session_id=session_id, citations=True)
43 | citations = []
44 | for i in metadata:
45 | citations.append(i[1]["url"])
46 | response = {"answer": answer, "citations": list(set(citations))}
47 | except Exception as e:
48 | response = f"An error occurred: Error message: {str(e)}. Contact Embedchain founders on Slack: https://embedchain.com/slack or Discord: https://embedchain.com/discord" # noqa:E501
49 | return {"response": response}
50 |
51 |
52 | @router.get("/api/v1/channels")
53 | async def get_channels():
54 | """
55 | Handles a channel list request to the Embedchain app.
56 | Accepts 'user_token' as a query parameter.
57 | """
58 | try:
59 | # List all channels in slack: https://api.slack.com/methods/conversations.list
60 | response = loader.get_channels(os.environ["SLACK_USER_TOKEN"])
61 | return {"message": response}
62 | except Exception as e:
63 | response = f"An error occurred: Error message: {str(e)}. Contact Embedchain founders on Slack: https://embedchain.com/slack or Discord: https://embedchain.com/discord" # noqa:E501
64 | return {"message": response}
65 |
66 |
67 | @router.get("/api/v1/reset")
68 | async def reset_app():
69 | """
70 | Resets the Embedchain app.
71 | """
72 | try:
73 | ec_app.reset()
74 | return {"message": "Embedchain app reset successfully."}
75 | except Exception:
76 | response = "An error occurred: Error message: {str(e)}. Contact Embedchain founders on Slack: https://embedchain.com/slack or Discord: https://embedchain.com/discord" # noqa:E501
77 | return {"message": response}
78 |
79 |
80 | @router.get("/")
81 | async def root():
82 | return responses.RedirectResponse(url="/docs")
83 |
--------------------------------------------------------------------------------
/slack-ai/api/utils/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/mem0ai/examples/f33f9c347abdab3a18d833f8392cbb4f7643d2ab/slack-ai/api/utils/__init__.py
--------------------------------------------------------------------------------
/slack-ai/api/utils/app.py:
--------------------------------------------------------------------------------
1 | from dotenv import load_dotenv
2 | from embedchain import App
3 | from embedchain.models.data_type import DataType
4 |
5 | from .slack_chunker import SlackChunker
6 | from .slack_loader import SlackLoader
7 |
8 | load_dotenv(".env")
9 |
10 | loader = SlackLoader()
11 | chunker = SlackChunker()
12 | chunker.set_data_type(DataType.SLACK)
13 |
14 | # Uncomment this line if you want to use HuggingFace LLMs
15 | app_config = {
16 | "app": {
17 | "config": {
18 | "name": "slack-ai"
19 | }
20 | },
21 | "llm": {
22 | "provider": "huggingface",
23 | "config": {
24 | "model": "mistralai/Mixtral-8x7B-Instruct-v0.1",
25 | "temperature": 0.1,
26 | "max_tokens": 250,
27 | "top_p": 0.1
28 | }
29 | },
30 | "embedder": {
31 | "provider": "huggingface",
32 | "config": {
33 | "model": "sentence-transformers/all-mpnet-base-v2"
34 | }
35 | }
36 | }
37 |
38 | # app_config = {
39 | # "app": {
40 | # "config": {
41 | # "id": "slack-ai-app",
42 | # }
43 | # },
44 | # "llm": {
45 | # "provider": "openai",
46 | # "config": {
47 | # "model": "gpt-3.5-turbo-1106",
48 | # },
49 | # },
50 | # }
51 |
52 |
53 | ec_app = App.from_config(config=app_config)
54 |
--------------------------------------------------------------------------------
/slack-ai/api/utils/slack_chunker.py:
--------------------------------------------------------------------------------
1 | from embedchain.chunkers.base_chunker import BaseChunker
2 |
3 |
4 | class SlackChunker(BaseChunker):
5 | def __init__(self):
6 | text_splitter = lambda x: x # noqa: E731
7 | super().__init__(text_splitter)
8 |
9 | def get_chunks(self, content):
10 | return [content]
--------------------------------------------------------------------------------
/slack-ai/api/utils/slack_loader.py:
--------------------------------------------------------------------------------
1 | import hashlib
2 | import logging
3 | import os
4 | import ssl
5 | from typing import Any, Dict, Optional
6 | from urllib.parse import parse_qs, urlparse
7 |
8 | import certifi
9 | from embedchain.loaders.base_loader import BaseLoader
10 | from embedchain.utils.misc import clean_string
11 |
12 |
13 | def get_thread_ts(url, parent_ts):
14 | parsed_url = urlparse(url)
15 | query_params = parse_qs(parsed_url.query)
16 | thread_ts = query_params.get("thread_ts", [None])[0]
17 | return thread_ts if thread_ts else parent_ts
18 |
19 |
20 | SLACK_API_BASE_URL = "https://www.slack.com/api/"
21 |
22 |
23 | class SlackLoader(BaseLoader):
24 | def __init__(self, config: Optional[Dict[str, Any]] = None):
25 | super().__init__()
26 |
27 | self.config = config if config else {}
28 |
29 | if "base_url" not in self.config:
30 | self.config["base_url"] = SLACK_API_BASE_URL
31 |
32 | self.client = None
33 | self._setup_loader(self.config)
34 |
35 | def _setup_loader(self, config: Dict[str, Any]):
36 | try:
37 | from slack_sdk import WebClient
38 | except ImportError as e:
39 | raise ImportError(
40 | "Slack loader requires extra dependencies. \
41 | Install with `pip install --upgrade embedchain[slack]`"
42 | ) from e
43 |
44 | if os.getenv("SLACK_USER_TOKEN") is None:
45 | raise ValueError(
46 | "SLACK_USER_TOKEN environment variables not provided. Check `https://docs.embedchain.ai/data-sources/slack` to learn more." # noqa:E501
47 | )
48 |
49 | logging.info(f"Creating Slack Loader with config: {config}")
50 | # get slack client config params
51 | slack_bot_token = os.getenv("SLACK_USER_TOKEN")
52 | ssl_cert = ssl.create_default_context(cafile=certifi.where())
53 | base_url = config.get("base_url", SLACK_API_BASE_URL)
54 | headers = config.get("headers")
55 | # for Org-Wide App
56 | team_id = config.get("team_id")
57 |
58 | self.client = WebClient(
59 | token=slack_bot_token,
60 | base_url=base_url,
61 | ssl=ssl_cert,
62 | headers=headers,
63 | team_id=team_id,
64 | )
65 | logging.info("Slack Loader setup successful!")
66 |
67 | def _check_query(self, query):
68 | if not isinstance(query, str):
69 | raise ValueError(
70 | f"Invalid query passed to Slack loader, found: {query}. Check `https://docs.embedchain.ai/data-sources/slack` to learn more." # noqa:E501
71 | )
72 |
73 | def load_data(self, query):
74 | self._check_query(query)
75 | try:
76 | message_data = []
77 | data = []
78 | data_content = []
79 |
80 | logging.info(f"Searching slack conversations for {query=}")
81 | results = self.client.search_messages(
82 | query=query,
83 | sort="timestamp",
84 | sort_dir="desc",
85 | count=self.config.get("count", 100),
86 | )
87 | messages = results.get("messages")
88 | num_message = messages.get("total")
89 | total_pages = messages.get("pagination").get("page_count")
90 | current_page = messages.get("pagination").get("page")
91 | print(f"Collecting {num_message} messages for {query=}, from {total_pages=}")
92 | message_data.extend(messages.get("matches", []))
93 | for page in range(current_page + 1, total_pages + 1):
94 | results = self.client.search_messages(
95 | query=query, sort="timestamp", sort_dir="desc", count=self.config.get("count", 100), page=page
96 | )
97 | messages = results.get("messages")
98 | message_data.extend(messages.get("matches", []))
99 |
100 | # group thread messages
101 | print("Grouping messages in threads...")
102 | message_threads = {}
103 | for message in message_data:
104 | url = message.get("permalink")
105 | text = message.get("text")
106 | content = clean_string(text)
107 |
108 | message_meta_data_keys = ["iid", "team", "ts", "type", "user", "username"]
109 | meta_data = {}
110 | for key in message.keys():
111 | if key in message_meta_data_keys:
112 | meta_data[key] = message.get(key)
113 | meta_data.update({"url": url})
114 | thread_ts = get_thread_ts(url, meta_data.get("ts"))
115 | if thread_ts not in message_threads:
116 | message_threads[thread_ts] = [(content, meta_data, meta_data.get("ts"))]
117 | else:
118 | message_threads[thread_ts].append((content, meta_data, meta_data.get("ts")))
119 |
120 | for url, messages in message_threads.items():
121 | messages = sorted(messages, key=lambda x: x[2])
122 | content = "\n".join([f"@{message[1].get('username')}: {message[0]}" for message in messages])
123 | meta_data = messages[0][1]
124 | data.append(
125 | {
126 | "content": content,
127 | "meta_data": meta_data,
128 | }
129 | )
130 | data_content.append(content)
131 |
132 | doc_id = hashlib.md5((query + ", ".join(data_content)).encode()).hexdigest()
133 | return {
134 | "doc_id": doc_id,
135 | "data": data,
136 | }
137 | except Exception as e:
138 | logging.warning(f"Error in loading slack data: {e}")
139 | raise ValueError(
140 | f"Error in loading slack data: {e}. Check `https://docs.embedchain.ai/components/data-sources/slack` to learn more." # noqa:E501
141 | ) from e
142 |
143 | def get_channels(self, user_token):
144 | try:
145 | results = self.client.conversations_list(
146 | token=user_token,
147 | type="public_channel",
148 | limit=100)
149 | channels = results.get("channels")
150 | public_channels = []
151 | for channel in channels:
152 | if channel.get("is_channel", False):
153 | public_channels.append(channel)
154 | return public_channels
155 | except Exception as e:
156 | logging.warning(f"Error in loading slack channels: {e}")
157 | raise ValueError(
158 | f"Error in loading slack channels: {e}. Check `https://docs.embedchain.ai/components/data-sources/slack` to learn more." # noqa:E501
159 | ) from e
160 |
--------------------------------------------------------------------------------
/slack-ai/docker-compose.yml:
--------------------------------------------------------------------------------
1 | version: '3.9'
2 |
3 | services:
4 | frontend:
5 | build:
6 | context: ui/
7 | dockerfile: Dockerfile
8 | volumes:
9 | - ./ui:/app
10 | - /app/node_modules
11 | ports:
12 | - "3000:3000"
13 | environment:
14 | - NODE_ENV=development
15 | - NEXT_PUBLIC_API_ENDPOINT=http://backend:8000
16 | depends_on:
17 | - backend
18 |
19 | backend:
20 | build:
21 | context: api/
22 | dockerfile: Dockerfile
23 | volumes:
24 | - ./api:/app
25 | env_file:
26 | - api/.env
27 | ports:
28 | - "8000:8000"
29 | environment:
30 | - PYTHONUNBUFFERED=1
31 |
--------------------------------------------------------------------------------
/slack-ai/ui/.env.local:
--------------------------------------------------------------------------------
1 | NEXT_PUBLIC_API_ENDPOINT=http://localhost:8000
2 |
--------------------------------------------------------------------------------
/slack-ai/ui/.eslintrc.json:
--------------------------------------------------------------------------------
1 | {
2 | "extends": "next/core-web-vitals"
3 | }
4 |
--------------------------------------------------------------------------------
/slack-ai/ui/.gitignore:
--------------------------------------------------------------------------------
1 | # See https://help.github.com/articles/ignoring-files/ for more about ignoring files.
2 |
3 | # dependencies
4 | /node_modules
5 | /.pnp
6 | .pnp.js
7 |
8 | # testing
9 | /coverage
10 |
11 | # next.js
12 | /.next/
13 | /out/
14 |
15 | # production
16 | /build
17 |
18 | # misc
19 | .DS_Store
20 | *.pem
21 |
22 | # debug
23 | npm-debug.log*
24 | yarn-debug.log*
25 | yarn-error.log*
26 |
27 | # vercel
28 | .vercel
29 |
30 | # typescript
31 | *.tsbuildinfo
32 | next-env.d.ts
33 |
--------------------------------------------------------------------------------
/slack-ai/ui/Dockerfile:
--------------------------------------------------------------------------------
1 | FROM node:latest
2 |
3 | WORKDIR /app
4 |
5 | COPY package.json yarn.lock ./
6 |
7 | RUN yarn install
8 |
9 | COPY . .
10 |
11 | EXPOSE 3000
12 |
13 | CMD ["yarn", "dev"]
14 |
--------------------------------------------------------------------------------
/slack-ai/ui/README.md:
--------------------------------------------------------------------------------
1 | This is Embedchain Admin repository.
2 |
3 | ## Getting Started
4 |
5 | First, run the development server:
6 |
7 | ```bash
8 | npm run dev
9 | # or
10 | yarn dev
11 | # or
12 | pnpm dev
13 | ```
14 |
15 | Open [http://localhost:3000](http://localhost:3000) with your browser to see the result.
16 |
17 | You can start editing the page by modifying `app/page.tsx`. The page auto-updates as you edit the file.
18 |
19 | This project uses [`next/font`](https://nextjs.org/docs/basic-features/font-optimization) to automatically optimize and load Inter, a custom Google Font.
20 |
21 | ## Learn More
22 |
23 | To learn more about Next.js, take a look at the following resources:
24 |
25 | - [Next.js Documentation](https://nextjs.org/docs) - learn about Next.js features and API.
26 | - [Learn Next.js](https://nextjs.org/learn) - an interactive Next.js tutorial.
27 |
28 | You can check out [the Next.js GitHub repository](https://github.com/vercel/next.js/) - your feedback and contributions are welcome!
29 |
30 | ## Deploy on Vercel
31 |
32 | The easiest way to deploy your Next.js app is to use the [Vercel Platform](https://vercel.com/new?utm_medium=default-template&filter=next.js&utm_source=create-next-app&utm_campaign=create-next-app-readme) from the creators of Next.js.
33 |
34 | Check out our [Next.js deployment documentation](https://nextjs.org/docs/deployment) for more details.
35 |
--------------------------------------------------------------------------------
/slack-ai/ui/app/admin/chromadb/collections/[collection_name]/page.tsx:
--------------------------------------------------------------------------------
1 | "use client";
2 |
3 | import { useState, useEffect } from "react";
4 | import { Maximize } from "lucide-react";
5 | import { Separator } from "@/components/ui/separator";
6 |
7 | import {
8 | Drawer,
9 | DrawerContent,
10 | DrawerHeader,
11 | DrawerTitle,
12 | DrawerTrigger,
13 | } from "@/components/ui/drawer";
14 |
15 | import {
16 | Table,
17 | TableBody,
18 | TableCell,
19 | TableHead,
20 | TableHeader,
21 | TableRow,
22 | } from "@/components/ui/table";
23 | import Link from "next/link";
24 |
25 | export default function Page({
26 | params,
27 | }: {
28 | params: { collection_name: string };
29 | }) {
30 | const [collectionData, setCollectionData] = useState([]);
31 | console.log(params);
32 |
33 | useEffect(() => {
34 | fetch(`/api/v1/admin/collections/chromadb/${params.collection_name}`)
35 | .then((response) => response.json())
36 | .then((data) => {
37 | setCollectionData(data);
38 | });
39 | }, []);
40 |
41 | return (
42 | Sources:
79 | {message.content.citations.map((citation, index) => ( 80 | 86 | {citation} 87 | 88 | ))} 89 |