├── .gitignore ├── CDB-Assignment ├── CrowdAssign │ ├── __init__.py │ ├── settings.py │ ├── urls.py │ └── wsgi.py ├── README.md ├── main │ ├── .gitkeep │ ├── __init__.py │ ├── admin.py │ ├── apps.py │ ├── models.py │ ├── property.py │ ├── task │ │ ├── __init__.py │ │ ├── amt.py │ │ ├── api_key.py │ │ ├── assign.py │ │ ├── cc.py │ │ ├── cf.py │ │ ├── chinacrowd.py │ │ ├── crowddb_crowdflower.py │ │ ├── crowddb_type_collection.py │ │ ├── crowddb_type_fillin.py │ │ ├── crowddb_type_multi_label.py │ │ ├── crowddb_type_one_label.py │ │ ├── crowddb_type_y_n.py │ │ ├── infer │ │ │ ├── EM.py │ │ │ └── __init__.py │ │ ├── mtc.py │ │ └── task_const.py │ ├── tests.py │ ├── urls.py │ └── views.py └── manage.py ├── CDB-Server ├── README.md ├── build.sh ├── crowdcore │ ├── pom.xml │ └── src │ │ └── main │ │ └── java │ │ └── com │ │ └── tsinghua │ │ └── dbgroup │ │ └── crowddb │ │ └── crowdcore │ │ ├── configs │ │ └── GlobalConfigs.java │ │ └── exceptions │ │ ├── CrowdDBException.java │ │ └── ExceptionMap.java ├── crowdexec │ ├── pom.xml │ └── src │ │ ├── main │ │ ├── java │ │ │ └── com │ │ │ │ └── tsinghua │ │ │ │ └── dbgroup │ │ │ │ └── crowddb │ │ │ │ └── crowdexec │ │ │ │ ├── gmodel │ │ │ │ ├── Graph.java │ │ │ │ ├── Main.java │ │ │ │ ├── Record.java │ │ │ │ ├── Result.java │ │ │ │ ├── Similarity_function.java │ │ │ │ └── Table.java │ │ │ │ ├── operator │ │ │ │ ├── BaseOperator.java │ │ │ │ ├── CollectOperator.java │ │ │ │ ├── CrowdEQOperator.java │ │ │ │ ├── CrowdGTOperator.java │ │ │ │ ├── CrowdGraphOperator.java │ │ │ │ ├── CrowdInOperator.java │ │ │ │ ├── CrowdJoinOperator.java │ │ │ │ ├── CrowdLTOperator.java │ │ │ │ ├── EQOperator.java │ │ │ │ ├── FillOperator.java │ │ │ │ ├── GTOperator.java │ │ │ │ ├── IOperator.java │ │ │ │ ├── JoinOperator.java │ │ │ │ ├── LTOperator.java │ │ │ │ ├── MultiLabel.java │ │ │ │ ├── OperatorStatus.java │ │ │ │ ├── ProjectOperator.java │ │ │ │ └── SingleLabel.java │ │ │ │ └── query │ │ │ │ ├── IQueryExecutor.java │ │ │ │ ├── IQueryManager.java │ │ │ │ ├── QueryExecutor.java │ │ │ │ ├── QueryManager.java │ │ │ │ ├── RelationOperator.java │ │ │ │ └── schema │ │ │ │ ├── HibernateSessionManager.java │ │ │ │ └── Query.java │ │ └── resources │ │ │ └── hibernate │ │ │ ├── hibernate.cfg.xml │ │ │ └── query.hbm.xml │ │ └── test │ │ └── java │ │ └── com │ │ └── tsinghua │ │ └── dbgroup │ │ └── crowddb │ │ └── crowdexec │ │ ├── operator │ │ └── CrowdEQOperatorTest.java │ │ ├── query │ │ └── schema │ │ │ └── QueryControllerTest.java │ │ └── table │ │ └── TableManagerTest.java ├── crowdplat │ ├── pom.xml │ └── src │ │ ├── main │ │ └── java │ │ │ └── com │ │ │ └── tsinghua │ │ │ └── dbgroup │ │ │ └── crowddb │ │ │ └── crowdplat │ │ │ ├── TaskManager.java │ │ │ ├── core │ │ │ ├── Question.java │ │ │ ├── QuestionBuilder.java │ │ │ ├── Task.java │ │ │ ├── TaskCategory.java │ │ │ ├── TaskConfigure.java │ │ │ ├── TaskPair.java │ │ │ ├── TaskStatus.java │ │ │ └── TaskType.java │ │ │ ├── engine │ │ │ ├── APEngine.java │ │ │ ├── ChinaCrowdsEngine.java │ │ │ └── ICrowdEngine.java │ │ │ ├── http │ │ │ ├── HttpRequest.java │ │ │ └── IHttpRequest.java │ │ │ ├── result │ │ │ ├── BaseResult.java │ │ │ └── impl │ │ │ │ ├── CollectionResult.java │ │ │ │ ├── ColumnsResult.java │ │ │ │ ├── JudgementResult.java │ │ │ │ └── OptionsResult.java │ │ │ └── schema │ │ │ ├── BaseSchema.java │ │ │ ├── ISchema.java │ │ │ ├── SchemaType.java │ │ │ └── impl │ │ │ ├── CollectionSchema.java │ │ │ ├── FillSchema.java │ │ │ ├── JudgementSchema.java │ │ │ ├── LabelSchema.java │ │ │ └── OptionsSchema.java │ │ └── test │ │ └── java │ │ └── com │ │ └── tsinghua │ │ └── dbgroup │ │ └── crowddb │ │ └── crowdplat │ │ ├── TaskManagerTest.java │ │ ├── engine │ │ └── APEngineTest.java │ │ └── http │ │ └── HttpRequestTest.java ├── crowdscheduler │ ├── pom.xml │ └── src │ │ ├── main │ │ ├── java │ │ │ └── com │ │ │ │ └── tsinghua │ │ │ │ └── dbgroup │ │ │ │ └── crowddb │ │ │ │ └── scheduler │ │ │ │ ├── Configs.java │ │ │ │ ├── Server.java │ │ │ │ ├── dispatcher │ │ │ │ └── QueryScheduler.java │ │ │ │ ├── server │ │ │ │ └── SocketServer.java │ │ │ │ ├── threads │ │ │ │ ├── ContinueQueryThread.java │ │ │ │ ├── FinishQueryThread.java │ │ │ │ └── NewQueryThread.java │ │ │ │ └── utils │ │ │ │ └── Util.java │ │ └── resources │ │ │ └── logback.xml │ │ └── test │ │ └── java │ │ └── com │ │ └── tsinghua │ │ └── dbgroup │ │ └── crowddb │ │ └── scheduler │ │ ├── dispatcher │ │ └── QuerySchedulerTest.java │ │ └── server │ │ └── SocketServerTest.java ├── crowdsql │ ├── pom.xml │ └── src │ │ ├── main │ │ └── java │ │ │ ├── META-INF │ │ │ └── MANIFEST.MF │ │ │ └── com │ │ │ └── tsinghua │ │ │ └── dbgroup │ │ │ └── crowddb │ │ │ └── crowdsql │ │ │ ├── operator │ │ │ ├── OperatorHelper.java │ │ │ └── Operators.java │ │ │ ├── parser │ │ │ ├── ISqlParser.java │ │ │ └── SqlParser.java │ │ │ ├── query │ │ │ └── SqlContext.java │ │ │ ├── tree │ │ │ ├── NodeType.java │ │ │ ├── SqlTree.java │ │ │ ├── SqlTreeBuilder.java │ │ │ └── SqlTreeNode.java │ │ │ └── util │ │ │ └── Utils.java │ │ └── test │ │ └── java │ │ └── com │ │ └── tsinghua │ │ └── dbgroup │ │ └── crowddb │ │ └── crowdsql │ │ ├── parser │ │ └── SqlParserTest.java │ │ └── query │ │ └── SqlContextTest.java ├── crowdstorage │ ├── pom.xml │ └── src │ │ └── main │ │ ├── java │ │ └── com │ │ │ └── tsinghua │ │ │ └── dbgroup │ │ │ └── crowddb │ │ │ └── crowdstorage │ │ │ ├── table │ │ │ ├── BaseDBStorage.java │ │ │ ├── ITableManager.java │ │ │ ├── TableManager.java │ │ │ └── schema │ │ │ │ ├── ColumnsSchema.java │ │ │ │ ├── EqualSchema.java │ │ │ │ └── JoinSchema.java │ │ │ └── utils │ │ │ ├── TableHelper.java │ │ │ └── Utils.java │ │ └── resources │ │ └── jdbc.properties.development ├── pom.xml ├── resources │ ├── META-INF │ │ └── MANIFEST.MF │ ├── logback.xml │ └── runtime.configs.properties ├── scripts │ ├── runing_scripts │ │ ├── init_db.sh │ │ └── init_db.sql │ └── test_scripts │ │ ├── send_request.py │ │ └── test_server.py └── start.sh ├── CDB-WEB ├── README.md ├── front │ ├── .babelrc │ ├── .editorconfig │ ├── .gitignore │ ├── README.md │ ├── build │ │ ├── build.js │ │ ├── check-versions.js │ │ ├── dev-client.js │ │ ├── dev-server.js │ │ ├── utils.js │ │ ├── webpack.base.conf.js │ │ ├── webpack.dev.conf.js │ │ └── webpack.prod.conf.js │ ├── config │ │ ├── dev.env.js │ │ ├── index.js │ │ └── prod.env.js │ ├── index.html │ ├── package.json │ ├── src │ │ ├── App.vue │ │ ├── assets │ │ │ ├── login.jpg │ │ │ └── logo.png │ │ ├── components │ │ │ ├── CrowdShow.vue │ │ │ └── Hello.vue │ │ ├── main.js │ │ └── store │ │ │ ├── actions.js │ │ │ ├── index.js │ │ │ └── mutation_types.js │ └── static │ │ └── .gitkeep └── service │ ├── README.md │ ├── app.py │ ├── checks.py │ ├── dbcontrol.py │ ├── error.py │ ├── model.py │ ├── remove_user.py │ ├── requirements.txt │ ├── static │ ├── css │ │ ├── app.8d53a3b4be94e580d5043f5e3d68dd4f.css │ │ └── app.8d53a3b4be94e580d5043f5e3d68dd4f.css.map │ ├── img │ │ └── login.abdb9ad.jpg │ └── js │ │ ├── app.fbf9f7e5f297ee9b64d6.js │ │ ├── app.fbf9f7e5f297ee9b64d6.js.map │ │ ├── manifest.dbb9dc97ad79361bbc8e.js │ │ ├── manifest.dbb9dc97ad79361bbc8e.js.map │ │ ├── vendor.fe49eeef8e49276803d9.js │ │ └── vendor.fe49eeef8e49276803d9.js.map │ ├── templates │ └── index.html │ ├── upload │ └── .gitignore │ ├── util.py │ └── web.py └── README.md /.gitignore: -------------------------------------------------------------------------------- 1 | *.iml 2 | *.pyc 3 | CDB-Assignment/main/migrations 4 | *.DS_Store 5 | CDB-Assignment/main/task/cf_temp/* 6 | CDB-Assignment/!main/task/cf_temp/.gitkeep 7 | .idea/ 8 | CDB-Server/target/ 9 | crowddb.iml 10 | *.log 11 | *.iml 12 | *.class 13 | *.jar 14 | CDB-Server/out/ 15 | -------------------------------------------------------------------------------- /CDB-Assignment/CrowdAssign/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TsinghuaDatabaseGroup/CDB/6d949d89247598d8785d59d25d0557726d4e652f/CDB-Assignment/CrowdAssign/__init__.py -------------------------------------------------------------------------------- /CDB-Assignment/CrowdAssign/urls.py: -------------------------------------------------------------------------------- 1 | """CrowdAssign URL Configuration 2 | 3 | The `urlpatterns` list routes URLs to views. For more information please see: 4 | https://docs.djangoproject.com/en/1.10/topics/http/urls/ 5 | Examples: 6 | Function views 7 | 1. Add an import: from my_app import views 8 | 2. Add a URL to urlpatterns: url(r'^$', views.home, name='home') 9 | Class-based views 10 | 1. Add an import: from other_app.views import Home 11 | 2. Add a URL to urlpatterns: url(r'^$', Home.as_view(), name='home') 12 | Including another URLconf 13 | 1. Import the include() function: from django.conf.urls import url, include 14 | 2. Add a URL to urlpatterns: url(r'^blog/', include('blog.urls')) 15 | """ 16 | from django.conf.urls import include, url 17 | from django.contrib import admin 18 | 19 | urlpatterns = [ 20 | url(r'^', include('main.urls')), 21 | url(r'^admin/', admin.site.urls), 22 | ] 23 | -------------------------------------------------------------------------------- /CDB-Assignment/CrowdAssign/wsgi.py: -------------------------------------------------------------------------------- 1 | """ 2 | WSGI config for CrowdAssign project. 3 | 4 | It exposes the WSGI callable as a module-level variable named ``application``. 5 | 6 | For more information on this file, see 7 | https://docs.djangoproject.com/en/1.10/howto/deployment/wsgi/ 8 | """ 9 | 10 | import os 11 | 12 | from django.core.wsgi import get_wsgi_application 13 | 14 | os.environ.setdefault("DJANGO_SETTINGS_MODULE", "CrowdAssign.settings") 15 | 16 | application = get_wsgi_application() 17 | -------------------------------------------------------------------------------- /CDB-Assignment/README.md: -------------------------------------------------------------------------------- 1 | ## CrowdAssign 2 | ### Introduction 3 | Assignment module of CrowdDB 4 | 5 | This module acts as a server to receive requests with well-formatted crowdsourcing tasks data from CDB-Server and publish the tasks to different platforms including AMT (Amazon Mechanical Turk), CF (CrowdFlower), CC (Chinacrowds). 6 | 7 | 8 | ### Environment Specs 9 | This module has been tested on MySQL 5.6.19, Python 2.7.12 with packages below: 10 | 11 | |Package|Version| 12 | |:-----:|:-----:| 13 | |Django | 1.10.1| 14 | |boto | 2.43.0| 15 | |MySQL-python|1.2.5| 16 | |crowdflower|0.1.4| 17 | |poster|0.8.1| 18 | 19 | ### Setup 20 | #### Software install 21 | First install **mysql** and **python**, then install required packages using commands like below: 22 | 23 | ``` 24 | pip install django 25 | pip install boto 26 | pip install mysql-python 27 | pip install crowdflower 28 | pip install poster 29 | ``` 30 | 31 | #### Configuration 32 | 1. Configure your MySQL connection parameters in **CrowdAssign/settings.py**, around line 78, in **DATABASE** dict, fill in your database name, username and password. 33 | 2. Use commands below to create corresponding tables in MySQL: 34 | ``` 35 | python manage.py makemigrations main 36 | python manage.py migrate 37 | ``` 38 | 3. Fill in your API-KEY or username/password and switch to production/test environment in corresponding file in: 39 | 40 | |Platform| File | 41 | |:------:|:----:| 42 | | AMT |main/task/mtc.py| 43 | |CrowdFlower|main/task/api_key.py| 44 | |Chinacrowds|main/property.py| 45 | 46 | 47 | 4. Run server (listen on localhost:9000) to see if everything goes fine. 48 | ``` 49 | python manage.py runserver 9000 50 | ``` 51 | -------------------------------------------------------------------------------- /CDB-Assignment/main/.gitkeep: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TsinghuaDatabaseGroup/CDB/6d949d89247598d8785d59d25d0557726d4e652f/CDB-Assignment/main/.gitkeep -------------------------------------------------------------------------------- /CDB-Assignment/main/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TsinghuaDatabaseGroup/CDB/6d949d89247598d8785d59d25d0557726d4e652f/CDB-Assignment/main/__init__.py -------------------------------------------------------------------------------- /CDB-Assignment/main/admin.py: -------------------------------------------------------------------------------- 1 | from django.contrib import admin 2 | 3 | # Register your models here. 4 | -------------------------------------------------------------------------------- /CDB-Assignment/main/apps.py: -------------------------------------------------------------------------------- 1 | from __future__ import unicode_literals 2 | 3 | from django.apps import AppConfig 4 | 5 | 6 | class MainConfig(AppConfig): 7 | name = 'main' 8 | -------------------------------------------------------------------------------- /CDB-Assignment/main/models.py: -------------------------------------------------------------------------------- 1 | ### table models definition, save tasks, as well as questions and answers info 2 | from __future__ import unicode_literals 3 | 4 | from django.db import models 5 | 6 | 7 | class Task(models.Model): 8 | PLATFORMS = ( 9 | ('AMT', 'Amazon Mechanical Turk'), 10 | ('CF', 'CrowdFlower'), 11 | ('CC', 'ChinaCrowd'), 12 | ) 13 | Q_TYPES = ( 14 | ('Y_N', 'Yes or No'), 15 | ('M_TO_O', 'Many to One'), 16 | ('M_TO_M', 'Many to Many'), 17 | ('FREE', 'Free text'), 18 | ('COLLECT', 'Collection'), 19 | ) 20 | task_id = models.CharField(max_length=20, primary_key=True) 21 | platform = models.CharField(max_length=10, choices=PLATFORMS) 22 | q_type = models.CharField(max_length=10, choices=Q_TYPES) 23 | created_time = models.DateTimeField(auto_now_add=True) 24 | 25 | 26 | class Hit(models.Model): 27 | hit_id = models.CharField(max_length=50, primary_key=True) 28 | task = models.ForeignKey(Task) 29 | 30 | 31 | class Question(models.Model): 32 | id = models.CharField(max_length=50, primary_key=True) 33 | task = models.ForeignKey(Task) 34 | content = models.CharField(max_length=500) 35 | sequence = models.IntegerField() 36 | 37 | 38 | class Answer(models.Model): 39 | question = models.ForeignKey(Question) 40 | worker = models.CharField(max_length=50) 41 | answer = models.IntegerField() 42 | 43 | 44 | class TaskProject(models.Model): 45 | task_db = models.CharField(max_length=20) 46 | task_bao = models.CharField(max_length=200) 47 | 48 | 49 | class QueTask(models.Model): 50 | task_id = models.IntegerField() 51 | que_id = models.TextField() 52 | attr_id = models.TextField() 53 | unit_num = models.IntegerField(default=1) 54 | need_url = models.CharField(max_length=20, default="no") 55 | -------------------------------------------------------------------------------- /CDB-Assignment/main/property.py: -------------------------------------------------------------------------------- 1 | #encoding:utf-8 2 | class ChinaCrowd_Property: 3 | #chinacrowd_api 4 | # Api_Crowd = "http://www.crowdbao.com:6789"; # product 5 | Api_Crowd = "http://166.111.71.172:6789" # test 6 | name = "Your Chinacrowds username" 7 | password = "Your Chinacrowds passowrd" -------------------------------------------------------------------------------- /CDB-Assignment/main/task/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TsinghuaDatabaseGroup/CDB/6d949d89247598d8785d59d25d0557726d4e652f/CDB-Assignment/main/task/__init__.py -------------------------------------------------------------------------------- /CDB-Assignment/main/task/api_key.py: -------------------------------------------------------------------------------- 1 | 2 | 3 | api_key_string = 'Your api_key' # CrowdFlower API_KEY -------------------------------------------------------------------------------- /CDB-Assignment/main/task/assign.py: -------------------------------------------------------------------------------- 1 | ### assign tasks to different platforms 2 | from main.models import Task 3 | from infer import EM 4 | from task_const import * 5 | 6 | def get_task_publisher(taskid): 7 | task = Task.objects.get(task_id=taskid) 8 | return platforms.get(task.platform, amt) 9 | 10 | 11 | def alter_questions(option, questions): 12 | if option['q_type'] == COLLECT: 13 | questions[0]['repeats'] = 2 14 | 15 | 16 | def publish(option, questions, platform): 17 | platform = platform.strip() 18 | if platform not in platforms: 19 | platform = default_platform 20 | Task.objects.create(task_id=option['task_id'], platform=platform, q_type=option['q_type']) 21 | publisher = platforms.get(platform, amt) 22 | alter_questions(option, questions) 23 | publisher.publish(option, questions) 24 | 25 | 26 | def is_complete(taskid): 27 | return get_task_publisher(taskid).is_complete(taskid) 28 | 29 | 30 | def get_all_result(taskid, q_type): 31 | return get_task_publisher(taskid).get_result(taskid, q_type) 32 | 33 | 34 | def get_aggregated_result(taskid): 35 | task = Task.objects.get(pk=taskid) 36 | result = get_all_result(taskid, task.q_type) 37 | if task.q_type == Y_N: 38 | result = EM.infer(result) 39 | for i in result: 40 | result[i] = int(result[i]) 41 | result_dict = {} 42 | for i in result: 43 | result_dict[i] = {'answer': result[i], 'id': i} 44 | result = result_dict 45 | elif task.q_type == M_TO_O: 46 | result = EM.infer(result) 47 | result_dict = {} 48 | for i in result: 49 | result_dict[i] = {'answer': [result[i]], 'id': i} 50 | result = result_dict 51 | elif task.q_type == FREE: 52 | result_dict = {} 53 | for line in result: 54 | # line: [q, w, a] 55 | result_dict[line[0]] = {'answer': line[2], 'id': line[0]} 56 | result = result_dict 57 | elif task.q_type == M_TO_M: 58 | result_dict = {} 59 | for line in result: 60 | result_dict[line[0]] = {'answer': line[2], 'id': line[0]} 61 | result = result_dict 62 | elif task.q_type == COLLECT: 63 | result_dict = {} 64 | qid = result[0][0].rsplit(free_sep, 1)[0] 65 | answer = [] 66 | for line in result: 67 | answer.append(line[2]) 68 | result_dict[qid] = {'answer': answer, 'id': qid} 69 | result = result_dict 70 | 71 | return result 72 | 73 | 74 | if __name__ == '__main__': 75 | # option = {'q_per_hit': 2} 76 | # questions = [ 77 | # { 78 | # 'id': 1, 79 | # 'content': 'First question' 80 | # }, 81 | # { 82 | # 'id': 2, 83 | # 'content': 'Second Question' 84 | # }, 85 | # { 86 | # 'id': 3, 87 | # 'content': 'Third with new line----\n\n\n----three times' 88 | # } 89 | # ] 90 | # 91 | # publish(option, questions, "AMT") 92 | get_all_result('1010') 93 | 94 | -------------------------------------------------------------------------------- /CDB-Assignment/main/task/cc.py: -------------------------------------------------------------------------------- 1 | ### chinacrowd interface 2 | import chinacrowd 3 | 4 | api = chinacrowd.ChinaBaoApi() 5 | 6 | 7 | def publish(option, questions): 8 | api.TaskCreated(questions, option['task_id'], option['q_type'], option['title']) 9 | 10 | 11 | def is_complete(taskid): 12 | return api.TaskDetected(taskid) 13 | 14 | 15 | def get_result(taskid, q_type): 16 | return api.TaskResult(taskid, q_type) -------------------------------------------------------------------------------- /CDB-Assignment/main/task/cf.py: -------------------------------------------------------------------------------- 1 | ### crowdflower interface 2 | import crowddb_crowdflower 3 | 4 | 5 | def publish(option, questions): 6 | crowddb_crowdflower.design_question(option['q_type'], option['task_id'], questions) 7 | 8 | 9 | def is_complete(taskid): 10 | return crowddb_crowdflower.query_the_status_of_the_job(taskid) 11 | 12 | 13 | def get_result(taskid, q_type): 14 | return crowddb_crowdflower.get_answers(taskid, q_type) 15 | -------------------------------------------------------------------------------- /CDB-Assignment/main/task/crowddb_type_one_label.py: -------------------------------------------------------------------------------- 1 | 2 | import os 3 | import sys 4 | sys.path.append("..") 5 | import csv 6 | import crowdflower 7 | import api_key 8 | 9 | def upload_questions_to_crowdflower(title, job_distinguish_tag, question_content, max_judgments_per_worker = 50, units_per_assignment = 3, 10 | judgments_per_unit = 1, payment_cents = 3): 11 | 12 | conn = crowdflower.Connection(api_key = api_key.api_key_string) 13 | job = conn.upload(question_content) 14 | options = question_content[0]['options'] 15 | first_line = ''' 16 | 17 |

{{content}}

18 | 19 | ''' 20 | question_strings = first_line 21 | for i in range(len(options)): 22 | item = '''''' 23 | question_strings = question_strings + item + '\n' 24 | question_strings = question_strings + '''''' 25 | 26 | job.update({ 27 | 'title': title, 28 | 'max_judgments_per_worker': max_judgments_per_worker, 29 | 'units_per_assignment': units_per_assignment, 30 | 'judgments_per_unit': judgments_per_unit, 31 | 'payment_cents': payment_cents, 32 | 'instructions': 33 | ''' 34 |

Tell us the details about the country 

35 | ''', 36 | 'cml': 37 | question_strings 38 | , 39 | 'options': { 40 | 'front_load': 0, # quiz mode = 1; turn off with 0 41 | } 42 | }) 43 | job.tags = [job_distinguish_tag] 44 | job.launch(len(question_content), channels = ('on_demand', 'cf_internal')) 45 | 46 | def collect_answers_from_crowdflower(job_distinguish_tag): 47 | conn = crowdflower.Connection(api_key=api_key.api_key_string) 48 | result = [] 49 | out_dir = os.path.dirname(os.path.realpath(__file__)) + os.path.sep + 'cf_temp/' 50 | for job in conn.jobs(): 51 | # if job_distinguish_tag in job.tags and job.properties['state'] == 'finished': 52 | if job_distinguish_tag in job.tags: 53 | job.download_csv(out_dir + str(job.id) + '.csv') 54 | reader = csv.reader(file(out_dir + str(job.id) + '.csv', 'rb')) 55 | first_line = True 56 | for line in reader: 57 | if first_line == True: 58 | first_line = False 59 | for index, item in enumerate(line): 60 | if item == 'id': 61 | id_index = index 62 | if item == 'category': 63 | category_index = index 64 | 65 | else: 66 | worker_id = line[7] 67 | unique_id = line[id_index] 68 | option_single_answer = line[category_index].split('_')[1] 69 | result.append((unique_id, worker_id, option_single_answer)) 70 | break 71 | return job_distinguish_tag, result 72 | 73 | if __name__ == "__main__": 74 | question_content = [ 75 | {'id': '1', 'content': 'Germany', 'url': 'http://farm9.staticflickr.com/8199/8252746471_7cd4cccc3b_n.jpg', 'options':['A', 'B', 'C']}, 76 | {'id': '2', 'content': 'China', 'url':'http://farm8.staticflickr.com/7287/8745135210_b556f8f586_n.jpg', 'options':['A', 'B', 'C']}, 77 | {'id': '3', 'content': 'USA', 'url':'http://farm9.staticflickr.com/8195/8093245580_d7c95a2eca_n.jpg', 'options':['A', 'B', 'C']} 78 | ] 79 | job_distinguish_tag = 'test-a-1' 80 | # collect_answers_from_crowdflower(job_distinguish_tag) 81 | upload_questions_to_crowdflower(job_distinguish_tag, question_content) 82 | 83 | 84 | 85 | -------------------------------------------------------------------------------- /CDB-Assignment/main/task/crowddb_type_y_n.py: -------------------------------------------------------------------------------- 1 | 2 | import os 3 | import csv 4 | import time 5 | import crowdflower 6 | import api_key 7 | 8 | def upload_questions_to_crowdflower(title, job_distinguish_tag, question_content, max_judgments_per_worker = 50, units_per_assignment = 3, 9 | judgments_per_unit = 1, payment_cents = 3): 10 | conn = crowdflower.Connection(api_key = api_key.api_key_string) 11 | job = conn.upload(question_content) 12 | job.update({ 13 | 'title': title, 14 | 'max_judgments_per_worker': max_judgments_per_worker, 15 | 'units_per_assignment': units_per_assignment, 16 | 'judgments_per_unit': judgments_per_unit, 17 | 'payment_cents': payment_cents, 18 | 'instructions': 19 | ''' 20 |

Semantic Discrimination 

21 |

Judge whether the two paragraphs describe the same object or not.

22 | ''', 23 | 'cml': 24 | ''' 25 |

{{content}}

26 | 27 | 28 | 29 | 30 | ''', 31 | 'options': { 32 | 'front_load': 0, # quiz mode = 1; turn off with 0 33 | } 34 | }) 35 | job.tags = [job_distinguish_tag] 36 | job.launch(len(question_content), channels = ('on_demand', 'cf_internal')) 37 | 38 | def collect_answers_from_crowdflower(job_distinguish_tag): 39 | conn = crowdflower.Connection(api_key = api_key.api_key_string) 40 | rating_result = [] 41 | out_dir = os.path.dirname(os.path.realpath(__file__)) + os.path.sep + 'cf_temp/' 42 | for job in conn.jobs(): 43 | if job_distinguish_tag in job.tags: 44 | job.download_csv(out_dir + str(job.id) + '.csv') 45 | reader = csv.reader(file(out_dir + str(job.id) + '.csv', 'rb')) 46 | first_line = True 47 | for line in reader: 48 | for index, item in enumerate(line): 49 | print index, item 50 | if item == 'id': 51 | id_index = index 52 | if item == 'option': 53 | option_index = index 54 | task_id = line[id_index] 55 | option = line[option_index] 56 | if option == 'same': 57 | option_result = 1 58 | elif option == 'different': 59 | option_result = 0 60 | worker_id = line[7] 61 | if first_line == False: 62 | rating_result.append([task_id, worker_id, option_result]) 63 | print [task_id, worker_id, option_result] 64 | else: 65 | first_line = False 66 | return job_distinguish_tag, rating_result 67 | 68 | -------------------------------------------------------------------------------- /CDB-Assignment/main/task/infer/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TsinghuaDatabaseGroup/CDB/6d949d89247598d8785d59d25d0557726d4e652f/CDB-Assignment/main/task/infer/__init__.py -------------------------------------------------------------------------------- /CDB-Assignment/main/task/mtc.py: -------------------------------------------------------------------------------- 1 | ### AMT configuration 2 | from django.conf import settings 3 | from boto.mturk.connection import MTurkConnection 4 | 5 | if True: # SANDBOX: 6 | HOST="mechanicalturk.sandbox.amazonaws.com" 7 | else: # PRODUCTION 8 | HOST="mechanicalturk.amazonaws.com" 9 | 10 | AWS_ACCESS_ID = 'Your AWS_ACCESS_ID' 11 | AWS_SECRET_KEY = 'Your AWS_SECRET_KEY' 12 | 13 | mtc = MTurkConnection(aws_access_key_id=AWS_ACCESS_ID, aws_secret_access_key=AWS_SECRET_KEY, host=HOST) 14 | -------------------------------------------------------------------------------- /CDB-Assignment/main/task/task_const.py: -------------------------------------------------------------------------------- 1 | Y_N = 'Y_N' 2 | M_TO_O = 'M_TO_O' 3 | M_TO_M = 'M_TO_M' 4 | FREE = 'FREE' 5 | COLLECT = 'COLLECT' 6 | 7 | import amt 8 | import cf 9 | import cc 10 | 11 | platforms = { 12 | "AMT": amt, 13 | "CC": cc, 14 | "CF": cf 15 | } 16 | 17 | default_platform = 'AMT' 18 | 19 | free_sep = '::' 20 | -------------------------------------------------------------------------------- /CDB-Assignment/main/tests.py: -------------------------------------------------------------------------------- 1 | from django.test import TestCase 2 | 3 | # Create your tests here. 4 | 5 | questions = [ 6 | { 7 | 'id': 1, 8 | 'content': 'First question', 9 | 'fields': ['first', 'second'], 10 | }, 11 | { 12 | 'id': 2, 13 | 'content': 'Second Question', 14 | 'fields': ['first', 'second'], 15 | }, 16 | { 17 | 'id': 3, 18 | 'content': 'Third with new line----\n\n\n----three times', 19 | 'fields': ['first', 'second'], 20 | } 21 | ] 22 | 23 | 24 | collect_test = { 25 | 'option': {'q_type': 'COLLECT', 'task_id': 'collect_3', 'title': 'collect_test_1'}, 26 | 'questions': [{ 27 | "id": "q-1-op-1-hit-e856e60g", 28 | "columns": [ 29 | "name", 30 | "school", 31 | "birthday" 32 | ], 33 | "limit": 3, 34 | "content": "Please judge whether 450 is equal to 450 ?" 35 | }] 36 | } -------------------------------------------------------------------------------- /CDB-Assignment/main/urls.py: -------------------------------------------------------------------------------- 1 | from django.conf.urls import url 2 | 3 | from . import views 4 | 5 | urlpatterns = [ 6 | url(r'^$', views.index, name='index'), 7 | url(r'^upload/$', views.upload), 8 | url(r'^check/$', views.check), 9 | url(r'^results/$', views.results), 10 | ] 11 | 12 | -------------------------------------------------------------------------------- /CDB-Assignment/main/views.py: -------------------------------------------------------------------------------- 1 | ### http requests handlers 2 | from django.shortcuts import render 3 | from django.http import HttpResponse, JsonResponse 4 | import random 5 | import json 6 | from task import assign 7 | import traceback 8 | 9 | test_tasks = {} 10 | 11 | 12 | def index(request): 13 | return HttpResponse('works!') 14 | 15 | 16 | def upload(request): 17 | data = json.loads(request.POST['data']) 18 | option = data['options'] 19 | questions = data['questions'] 20 | test_tasks[option['task_id']] = {'option': option, 'questions': questions} 21 | res = {} 22 | print option, questions 23 | platform = option['platform'] 24 | code = 1 25 | try: 26 | assign.publish(option, questions, platform) 27 | code = 0 28 | except Exception as e: 29 | print '[[ERROR]] -----upload-----' 30 | print option 31 | traceback.print_exc() 32 | res['code'] = code 33 | return JsonResponse(res, safe=False) 34 | 35 | 36 | def check(request): 37 | task_id = request.GET['task_id'] 38 | code = 1 39 | try: 40 | status = assign.is_complete(task_id) 41 | code = 0 42 | except Exception as e: 43 | print '[[ERROR]] -----check-----: ', task_id 44 | traceback.print_exc() 45 | res = {'code': code} 46 | if code == 0: 47 | res['status'] = status 48 | return JsonResponse(res, safe=False) 49 | 50 | 51 | def results(request): 52 | task_id = request.GET['task_id'] 53 | code = 1 54 | try: 55 | task_results = assign.get_aggregated_result(task_id) 56 | code = 0 57 | except Exception as e: 58 | print '[[ERROR]] -----results-----: ', task_id 59 | traceback.print_exc() 60 | res = {'code': code} 61 | if code == 0: 62 | res['data'] = task_results 63 | print task_results 64 | 65 | return JsonResponse(res, safe=False) 66 | -------------------------------------------------------------------------------- /CDB-Assignment/manage.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | import os 3 | import sys 4 | 5 | if __name__ == "__main__": 6 | os.environ.setdefault("DJANGO_SETTINGS_MODULE", "CrowdAssign.settings") 7 | try: 8 | from django.core.management import execute_from_command_line 9 | except ImportError: 10 | # The above import may fail for some other reason. Ensure that the 11 | # issue is really that Django is missing to avoid masking other 12 | # exceptions on Python 2. 13 | try: 14 | import django 15 | except ImportError: 16 | raise ImportError( 17 | "Couldn't import Django. Are you sure it's installed and " 18 | "available on your PYTHONPATH environment variable? Did you " 19 | "forget to activate a virtual environment?" 20 | ) 21 | raise 22 | execute_from_command_line(sys.argv) 23 | -------------------------------------------------------------------------------- /CDB-Server/README.md: -------------------------------------------------------------------------------- 1 | # CDB-Server 2 | 3 | ## Introduction 4 | 5 | CDB-Server is the kernal of CDB database, and it has 6 parts: 6 | ##### crowdcore 7 | It defines some core data structures and core componments. 8 | 9 | ##### crowdexec 10 | crowdexec is responsible for operator execution details. It will pack up operators into questions and interactive with CDB-Assignment platform. 11 | 12 | ##### crowdplat 13 | crowdplat is an interface between CDB-Server and CDB-platform. 14 | 15 | ##### crowdschedular 16 | crowdschedular receive queries from clients and append them into schedula queue. In every cycle, crowdschedular will check every query if its current 17 | operator has been finished. If it has been finished, crowdschedular stash the current operator's data and continue to the next operator. 18 | 19 | ##### crowdsql 20 | crowdsql will recevice an input query and parse it to a synax tree or graphmodel. 21 | 22 | ##### crowdstorage 23 | crowdstorage is responsible for table storage and intermediate data stash. 24 | 25 | ## Install 26 | 27 | ### Install Dependences 28 | CDB-Server is based on Ubuntu and developed by Java. We take `Maven` as the integrated build tools, and use `Mysql` as storage warehouse. 29 | 30 | Dependences: 31 | - Java: version 1.8.0 32 | - Mysql: version 14.14 33 | - Apache Maven: version 3.3.9 34 | - Python 2: version 2.7 35 | - Git: version 2.10.1 36 | 37 | Please install those dependences following by its official documents. 38 | 39 | ### Config and Init DB 40 | This step will init the database and create nesscery databases, users and tables in it. 41 | 42 | open `CDB-Server/scripts/running_scripts/init_db.sh` and edit `host`, `port`, `user` and `password`. 43 | ``` 44 | HOST="127.0.0.1" 45 | PORT="3306" 46 | USER="root" 47 | PASSWORD="" 48 | ``` 49 | 50 | execute the following command 51 | ``` 52 | sh CDB-Server/scripts/running_scripts/init_db.sh 53 | ``` 54 | 55 | ### Edit CDB-Server configs 56 | Go to `CDB-Server/resources/runtime.configs.properties` directory and edit the configs. Here listing some important configs. 57 | 58 | ``` 59 | # assignment server 60 | ASSIGNMENT_SERVER_URL = http://127.0.0.1:9000 61 | # server listening port 62 | PORT = 1234 63 | # The time interval of rolling queries 64 | POLL_INTERVAL = 300 65 | 66 | ### Crowdsourcing Platform Settings 67 | # the default crowdsourcing platform for answering questions, there are three options: 68 | # 1. CC = ChinaCrowds, http://chinacrowds.com/ 69 | # 2. CF = Crowdflower, http://www.crowdflower.com/ 70 | # 3. AMT = Amazon Mechanical Turk, https://www.mturk.com/mturk/welcome 71 | DEFAULT_PLATFORM = CC 72 | ``` 73 | 74 | ### Build Project 75 | Go to the `CDB-Server` directory and run the following command. 76 | ``` 77 | sh build.sh 78 | ``` 79 | 80 | ### Run server 81 | ``` 82 | sh start.sh 83 | ``` 84 | 85 | 86 | ### Test 87 | 88 | edit `CDB-Server/scripts/test_scripts/send_request.py` and edit `host` and `port` for CDB-Server. 89 | ``` 90 | host = "127.0.0.1" 91 | port = 1234 92 | ``` 93 | 94 | run the following command to test server and check logs from `logs/crowddb-[year]-[month].log`. 95 | ``` 96 | python send_request.py [query_id] 97 | ``` 98 | please replace the query_id to real query_id 99 | 100 | 101 | ## Contact 102 | If you have any questions, please feel free to contact Xueping Weng(wxping715@gmail.com) 103 | -------------------------------------------------------------------------------- /CDB-Server/build.sh: -------------------------------------------------------------------------------- 1 | mvn clean 2 | mvn install -Dmaven.test.skip=true 3 | -------------------------------------------------------------------------------- /CDB-Server/crowdcore/pom.xml: -------------------------------------------------------------------------------- 1 | 3 | 4 | crowddb 5 | com.tsinghua.dbgroup.crowddb 6 | 0.01 7 | 8 | 4.0.0 9 | 10 | crowdcore 11 | jar 12 | 13 | crowdcore 14 | http://maven.apache.org 15 | 16 | 17 | UTF-8 18 | 19 | 20 | 21 | 22 | junit 23 | junit 24 | 3.8.1 25 | test 26 | 27 | 28 | 29 | -------------------------------------------------------------------------------- /CDB-Server/crowdcore/src/main/java/com/tsinghua/dbgroup/crowddb/crowdcore/configs/GlobalConfigs.java: -------------------------------------------------------------------------------- 1 | package com.tsinghua.dbgroup.crowddb.crowdcore.configs; 2 | 3 | import java.util.Properties; 4 | 5 | /** 6 | * Created by talus on 11/19/16. 7 | */ 8 | public class GlobalConfigs { 9 | public static Properties GlobalConfigs; 10 | } 11 | -------------------------------------------------------------------------------- /CDB-Server/crowdcore/src/main/java/com/tsinghua/dbgroup/crowddb/crowdcore/exceptions/CrowdDBException.java: -------------------------------------------------------------------------------- 1 | package com.tsinghua.dbgroup.crowddb.crowdcore.exceptions; 2 | 3 | /** 4 | * Created by talus on 11/19/16. 5 | */ 6 | public class CrowdDBException extends RuntimeException { 7 | 8 | public CrowdDBException(int errorCode) { 9 | super(ExceptionMap.ErrorMap.getOrDefault(errorCode, "unknown error")); 10 | } 11 | } 12 | -------------------------------------------------------------------------------- /CDB-Server/crowdcore/src/main/java/com/tsinghua/dbgroup/crowddb/crowdcore/exceptions/ExceptionMap.java: -------------------------------------------------------------------------------- 1 | package com.tsinghua.dbgroup.crowddb.crowdcore.exceptions; 2 | 3 | import java.util.HashMap; 4 | import java.util.Map; 5 | 6 | /** 7 | * Created by talus on 11/19/16. 8 | */ 9 | public class ExceptionMap { 10 | 11 | public static Map ErrorMap = new HashMap<>(); 12 | 13 | static { 14 | initErrorMap(); 15 | } 16 | 17 | private static void initErrorMap() { 18 | ErrorMap.put(0x100, "sql parse error"); 19 | ErrorMap.put(0x200, "database error"); 20 | ErrorMap.put(0x300, "crowdsourcing platform error"); 21 | ErrorMap.put(0x400, "operator execution error"); 22 | } 23 | } 24 | -------------------------------------------------------------------------------- /CDB-Server/crowdexec/pom.xml: -------------------------------------------------------------------------------- 1 | 3 | 4 | crowddb 5 | com.tsinghua.dbgroup.crowddb 6 | 0.01 7 | 8 | 4.0.0 9 | 10 | crowdexec 11 | jar 12 | 13 | crowdexec 14 | http://maven.apache.org 15 | 16 | 17 | UTF-8 18 | 19 | 20 | 21 | 22 | junit 23 | junit 24 | 4.12 25 | test 26 | 27 | 28 | com.tsinghua.dbgroup.crowddb 29 | crowdsql 30 | 0.01 31 | 32 | 33 | 34 | commons-dbutils 35 | commons-dbutils 36 | 1.6 37 | 38 | 39 | org.mariadb.jdbc 40 | mariadb-java-client 41 | 1.5.4 42 | 43 | 44 | mysql 45 | mysql-connector-java 46 | 6.0.4 47 | jar 48 | compile 49 | 50 | 51 | org.hibernate 52 | hibernate-core 53 | 5.2.3.Final 54 | 55 | 56 | com.tsinghua.dbgroup.crowddb 57 | crowdplat 58 | 0.01 59 | 60 | 61 | com.tsinghua.dbgroup.crowddb 62 | crowdstorage 63 | 0.01 64 | 65 | 66 | 67 | 68 | -------------------------------------------------------------------------------- /CDB-Server/crowdexec/src/main/java/com/tsinghua/dbgroup/crowddb/crowdexec/gmodel/Main.java: -------------------------------------------------------------------------------- 1 | package com.tsinghua.dbgroup.crowddb.crowdexec.gmodel; 2 | 3 | public class Main { 4 | // public static void main(String[] args) { 5 | // /** 6 | // String str=System.getProperty("user.dir"); 7 | // System.out.print(str); 8 | // File file = new File(str,"hhh.txt"); 9 | // try { 10 | // file.createNewFile(); 11 | // FileWriter fileWritter = new FileWriter(file.getName(),true); 12 | // fileWritter.write("jdhakjdh"); 13 | // fileWritter.flush(); 14 | // //BufferedWriter bufferWritter = new BufferedWriter(fileWritter); 15 | // //bufferWritter.write("ejwdkj\nkjd"); 16 | // //bufferWritter.close(); 17 | // } catch (IOException e) { 18 | // 19 | // } 20 | // */ 21 | // /**ArrayList> a=new ArrayList>(); 22 | // ArrayList> b=new ArrayList>(); 23 | // ArrayList temp=new ArrayList(); 24 | // temp.add(9);temp.add(2);a.add(temp); 25 | // ArrayList temp1=new ArrayList(); 26 | // temp1.add(a.get(0).get(0)); 27 | // b.add(temp1); 28 | // a.get(0).set(0,10); 29 | // System.out.print(b.get(0).get(0));*/ 30 | // //String[] sql={"country.school=prof.school","prof.name=paper.name","paper.title=cite.title"}; 31 | // //String[] froms={"country","prof","paper","cite"}; 32 | // String[] joins_in_sql={"University.Name=Researcher.Affiliation","Researcher.Name=Papers.Author","Papers.Title=Citation.Title"}; 33 | // String[] sql_selection={"Papers.Conference=sigmod"}; 34 | // String[] froms_temp=new String[10]; 35 | // String[] joins_temp=new String[10]; 36 | // String[] from_in_sql={"University","Researcher","Papers","Citation"}; 37 | // 38 | // 39 | // for(int i=0;i set1 = new HashSet(); 13 | Set set2 = new HashSet(); 14 | Set set_union = new HashSet(); 15 | Set set_insection = new HashSet(); 16 | set_union.clear(); 17 | set_insection.clear(); 18 | String[] str1_split={}; 19 | String[] str2_split={}; 20 | str1_split=str1.split(" "); 21 | str2_split=str2.split(" "); 22 | for(String token:str1_split){ 23 | set1.add(token); 24 | } 25 | for(String token:str2_split){ 26 | set2.add(token); 27 | } 28 | set_union.addAll(set1); 29 | set_union.addAll(set2); 30 | set_insection.addAll(set1); 31 | set_insection.retainAll(set2); 32 | //System.out.println(set_insection.size()); 33 | //System.out.println(set_union.size()); 34 | return (float) set_insection.size()/set_union.size(); 35 | } 36 | public float bigram(String str1, String str2) 37 | { 38 | int i; 39 | String str1_append="",str2_append=""; 40 | for(i=0;i %s",newTableName,from0,column0,column1); 76 | 77 | try { 78 | tmpTableManager.execSQL(sql); 79 | tmpTableManager.deleteTmpTable(from0); 80 | }catch (SQLException e){ 81 | LOG.info(e.getMessage()); 82 | e.printStackTrace(); 83 | } 84 | 85 | sqlTreeNode.setTableName(newTableName); 86 | return newTableName; 87 | } 88 | } 89 | -------------------------------------------------------------------------------- /CDB-Server/crowdexec/src/main/java/com/tsinghua/dbgroup/crowddb/crowdexec/operator/IOperator.java: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright (C) 2016-2016 by The Department of Computer Science and 3 | * Technology, Tsinghua University 4 | * 5 | * Redistribution of this file is permitted under the terms of 6 | * the BSD license. 7 | * 8 | * Author : XuepingWeng 9 | * Created : 10/13/16 11:10 AM 10 | * Modified : 11 | * Contact : wxping715@gmail.com 12 | */ 13 | 14 | package com.tsinghua.dbgroup.crowddb.crowdexec.operator; 15 | 16 | 17 | public interface IOperator { 18 | 19 | public boolean process(); 20 | 21 | public String finish(); 22 | } 23 | -------------------------------------------------------------------------------- /CDB-Server/crowdexec/src/main/java/com/tsinghua/dbgroup/crowddb/crowdexec/operator/JoinOperator.java: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright (C) 2016-2016 by The Department of Computer Science and 3 | * Technology, Tsinghua University 4 | * 5 | * Redistribution of this file is permitted under the terms of 6 | * the BSD license. 7 | * 8 | * Author : XuepingWeng 9 | * Created : 10/13/16 11:58 AM 10 | * Modified : 11 | * Contact : wxping715@gmail.com 12 | */ 13 | 14 | package com.tsinghua.dbgroup.crowddb.crowdexec.operator; 15 | 16 | import com.tsinghua.dbgroup.crowddb.crowdstorage.table.TableManager; 17 | import com.tsinghua.dbgroup.crowddb.crowdplat.core.TaskCategory; 18 | import com.tsinghua.dbgroup.crowddb.crowdplat.core.TaskType; 19 | import com.tsinghua.dbgroup.crowddb.crowdsql.tree.SqlTreeNode; 20 | import com.tsinghua.dbgroup.crowddb.crowdstorage.utils.Utils; 21 | import org.apache.commons.lang3.tuple.Pair; 22 | import org.slf4j.Logger; 23 | import org.slf4j.LoggerFactory; 24 | 25 | public class JoinOperator extends BaseOperator implements IOperator { 26 | private static Logger LOG = LoggerFactory.getLogger(JoinOperator.class); 27 | 28 | String from0,from1,column0,column1,DBNAME; 29 | 30 | TableManager tmpTableManager = new TableManager(); 31 | 32 | 33 | public JoinOperator(SqlTreeNode sqlTreeNode,String dbName) { 34 | super(sqlTreeNode); 35 | this.taskCategory = TaskCategory.TEXT; 36 | this.taskType = TaskType.CROWD_EQUAL; 37 | this.DBNAME= dbName; 38 | } 39 | 40 | 41 | private void initialFromColumn(){ 42 | 43 | String[] part = sqlTreeNode.getWhereClause().split(" "); 44 | 45 | from0= part[0].substring(0,part[0].indexOf(".")); 46 | 47 | column0 = part[0].substring(part[0].indexOf('.')+1,part[0].length()); 48 | 49 | from1 = part[2].substring(0,part[2].indexOf(".")); 50 | 51 | column1 = part[2].substring(part[2].indexOf('.')+1,part[2].length()); 52 | 53 | LOG.info(DBNAME); 54 | 55 | if (sqlTreeNode.getLeft()!=null) { 56 | 57 | column0 = Utils.packetColumn(from0, column0); 58 | from0 = sqlTreeNode.getLeft().getTableName(); 59 | } 60 | else { 61 | column0 = Utils.packetColumn(from0, column0); 62 | 63 | from0 = Utils.packetTable(DBNAME,from0); 64 | String newTableName = TableManager.TMP_DATABASE +"."+TableManager.generateTableName(); 65 | tmpTableManager.packetTable(newTableName,from0); 66 | from0 = newTableName; 67 | } 68 | column1 = Utils.packetColumn(from1,column1); 69 | 70 | if (sqlTreeNode.getRight()!=null) { 71 | from1 = sqlTreeNode.getRight().getTableName(); 72 | } 73 | else { 74 | from1 = Utils.packetTable(DBNAME,from1); 75 | String newTableName = TableManager.TMP_DATABASE +"."+TableManager.generateTableName(); 76 | tmpTableManager.packetTable(newTableName,from1); 77 | from1 = newTableName; 78 | } 79 | } 80 | @Override 81 | public boolean process() { 82 | initialFromColumn(); 83 | return true; 84 | } 85 | 86 | @Override 87 | public String finish() { 88 | String newTableName = TableManager.TMP_DATABASE+"."+TableManager.generateTableName(); 89 | tmpTableManager.normalJoinTables(newTableName,from0,from1,Pair.of(column0,column1), true); 90 | this.sqlTreeNode.setTableName(newTableName); 91 | return newTableName; 92 | } 93 | } 94 | -------------------------------------------------------------------------------- /CDB-Server/crowdexec/src/main/java/com/tsinghua/dbgroup/crowddb/crowdexec/operator/LTOperator.java: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright (C) 2016-2016 by The Department of Computer Science and 3 | * Technology, Tsinghua University 4 | * 5 | * Redistribution of this file is permitted under the terms of 6 | * the BSD license. 7 | * 8 | * Author : XuepingWeng 9 | * Created : 10/13/16 11:58 AM 10 | * Modified : 11 | * Contact : wxping715@gmail.com 12 | */ 13 | 14 | package com.tsinghua.dbgroup.crowddb.crowdexec.operator; 15 | 16 | import com.tsinghua.dbgroup.crowddb.crowdstorage.table.TableManager; 17 | import com.tsinghua.dbgroup.crowddb.crowdplat.core.TaskCategory; 18 | import com.tsinghua.dbgroup.crowddb.crowdplat.core.TaskType; 19 | import com.tsinghua.dbgroup.crowddb.crowdsql.tree.SqlTreeNode; 20 | import com.tsinghua.dbgroup.crowddb.crowdstorage.utils.Utils; 21 | import org.slf4j.Logger; 22 | import org.slf4j.LoggerFactory; 23 | 24 | import java.sql.SQLException; 25 | 26 | public class LTOperator extends BaseOperator implements IOperator { 27 | private static Logger LOG = LoggerFactory.getLogger(LTOperator.class); 28 | 29 | String from0,from1,column0,column1,DBNAME; 30 | 31 | TableManager tmpTableManager = new TableManager(); 32 | 33 | 34 | public LTOperator(SqlTreeNode sqlTreeNode,String dbName) { 35 | super(sqlTreeNode); 36 | this.taskCategory = TaskCategory.TEXT; 37 | this.taskType = TaskType.CROWD_EQUAL; 38 | this.DBNAME= dbName; 39 | } 40 | 41 | 42 | private void initialFromColumn(){ 43 | 44 | String[] part = sqlTreeNode.getWhereClause().split(" "); 45 | LOG.info(sqlTreeNode.getWhereClause()); 46 | LOG.info(part[0]); 47 | from0= part[0].substring(0,part[0].indexOf(".")); 48 | 49 | column0 = part[0].substring(part[0].indexOf('.')+1,part[0].length()); 50 | column1 = part[2]; 51 | 52 | 53 | column0 = Utils.packetColumn(from0, column0); 54 | if (sqlTreeNode.getLeft()!=null) { 55 | 56 | from0 = sqlTreeNode.getLeft().getTableName(); 57 | } 58 | else { 59 | 60 | from0 = Utils.packetTable(DBNAME ,from0); 61 | String newTableName = TableManager.TMP_DATABASE +"."+TableManager.generateTableName(); 62 | tmpTableManager.packetTable(newTableName,from0); 63 | from0 = newTableName; 64 | } 65 | } 66 | @Override 67 | public boolean process() { 68 | initialFromColumn(); 69 | return true; 70 | } 71 | 72 | @Override 73 | public String finish() { 74 | String newTableName = Utils.packetTable(TableManager.TMP_DATABASE, TableManager.generateTableName()); 75 | String sql=String.format("create table %s select * from %s where `%s` < %s",newTableName,from0,column0,column1); 76 | 77 | try { 78 | tmpTableManager.execSQL(sql); 79 | tmpTableManager.deleteTmpTable(from0); 80 | }catch (SQLException e){ 81 | LOG.info(e.getMessage()); 82 | e.printStackTrace(); 83 | } 84 | 85 | sqlTreeNode.setTableName(newTableName); 86 | return newTableName; 87 | } 88 | } 89 | -------------------------------------------------------------------------------- /CDB-Server/crowdexec/src/main/java/com/tsinghua/dbgroup/crowddb/crowdexec/operator/OperatorStatus.java: -------------------------------------------------------------------------------- 1 | package com.tsinghua.dbgroup.crowddb.crowdexec.operator; 2 | 3 | /** 4 | * Created by talus on 11/19/16. 5 | */ 6 | public enum OperatorStatus { 7 | /** 8 | * Have not sent task to crowd platform 9 | */ 10 | INIT, 11 | 12 | /** 13 | * Querying questions on crowd platform 14 | */ 15 | RUNNING, 16 | 17 | /** 18 | * Have received answers from platform, try to store into database 19 | */ 20 | FINISHING, 21 | 22 | /** 23 | * Current operator has finished 24 | */ 25 | FINISHED, 26 | } 27 | -------------------------------------------------------------------------------- /CDB-Server/crowdexec/src/main/java/com/tsinghua/dbgroup/crowddb/crowdexec/operator/ProjectOperator.java: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright (C) 2016-2016 by The Department of Computer Science and 3 | * Technology, Tsinghua University 4 | * 5 | * Redistribution of this file is permitted under the terms of 6 | * the BSD license. 7 | * 8 | * Author : XuepingWeng 9 | * Created : 10/13/16 11:58 AM 10 | * Modified : 11 | * Contact : wxping715@gmail.com 12 | */ 13 | 14 | package com.tsinghua.dbgroup.crowddb.crowdexec.operator; 15 | 16 | import com.tsinghua.dbgroup.crowddb.crowdstorage.table.TableManager; 17 | import com.tsinghua.dbgroup.crowddb.crowdplat.core.TaskCategory; 18 | import com.tsinghua.dbgroup.crowddb.crowdplat.core.TaskType; 19 | import com.tsinghua.dbgroup.crowddb.crowdsql.tree.SqlTreeNode; 20 | import com.tsinghua.dbgroup.crowddb.crowdstorage.utils.Utils; 21 | import org.slf4j.Logger; 22 | import org.slf4j.LoggerFactory; 23 | 24 | import java.sql.SQLException; 25 | import java.util.ArrayList; 26 | import java.util.List; 27 | 28 | public class ProjectOperator extends BaseOperator implements IOperator { 29 | private static Logger LOG = LoggerFactory.getLogger(ProjectOperator.class); 30 | 31 | String from0,from1,column0,column1,DBNAME; 32 | 33 | TableManager tmpTableManager = new TableManager(); 34 | 35 | 36 | public ProjectOperator(SqlTreeNode sqlTreeNode,String dbName) { 37 | super(sqlTreeNode); 38 | this.taskCategory = TaskCategory.TEXT; 39 | this.taskType = TaskType.CROWD_EQUAL; 40 | this.DBNAME= dbName; 41 | } 42 | 43 | 44 | private void initialFromColumn(){ 45 | // LOG.info(sqlTreeNode.getWhereClause()); 46 | from0= DBNAME+sqlTreeNode.getFroms().get(0); 47 | 48 | if (sqlTreeNode.getLeft()!=null) { 49 | from0 = sqlTreeNode.getLeft().getTableName(); 50 | }else{ 51 | String newTableName = TableManager.TMP_DATABASE +"."+TableManager.generateTableName(); 52 | tmpTableManager.packetTable(newTableName,from0); 53 | from0 = newTableName; 54 | } 55 | } 56 | @Override 57 | public boolean process() { 58 | initialFromColumn(); 59 | return true; 60 | } 61 | 62 | public String finish() { 63 | String newTableName = TableManager.generateTableName(); 64 | newTableName = Utils.packetTable(TableManager.TMP_DATABASE, newTableName); 65 | 66 | List fields = new ArrayList<>(); 67 | for (String col: sqlTreeNode.getProjects()) { 68 | fields.add(String.format("`%s`", col)); 69 | } 70 | String projects = String.join(", ", fields); 71 | 72 | String SQL=String.format("create table %s select %s from %s", newTableName, projects, from0); 73 | try { 74 | tmpTableManager.execSQL(SQL); 75 | tmpTableManager.deleteTmpTable(from0); 76 | }catch (SQLException e){ 77 | LOG.info(e.getMessage()); 78 | e.printStackTrace(); 79 | } 80 | 81 | sqlTreeNode.setTableName(newTableName); 82 | return newTableName; 83 | } 84 | } 85 | -------------------------------------------------------------------------------- /CDB-Server/crowdexec/src/main/java/com/tsinghua/dbgroup/crowddb/crowdexec/query/IQueryExecutor.java: -------------------------------------------------------------------------------- 1 | package com.tsinghua.dbgroup.crowddb.crowdexec.query; 2 | 3 | /** 4 | * Created by talus on 16/6/2. 5 | */ 6 | public interface IQueryExecutor { 7 | } 8 | -------------------------------------------------------------------------------- /CDB-Server/crowdexec/src/main/java/com/tsinghua/dbgroup/crowddb/crowdexec/query/IQueryManager.java: -------------------------------------------------------------------------------- 1 | package com.tsinghua.dbgroup.crowddb.crowdexec.query; 2 | 3 | /** 4 | * Created by talus on 16/6/2. 5 | */ 6 | public interface IQueryManager { 7 | 8 | } 9 | -------------------------------------------------------------------------------- /CDB-Server/crowdexec/src/main/java/com/tsinghua/dbgroup/crowddb/crowdexec/query/QueryExecutor.java: -------------------------------------------------------------------------------- 1 | package com.tsinghua.dbgroup.crowddb.crowdexec.query; 2 | 3 | import com.tsinghua.dbgroup.crowddb.crowdexec.operator.*; 4 | import com.tsinghua.dbgroup.crowddb.crowdsql.tree.NodeType; 5 | import com.tsinghua.dbgroup.crowddb.crowdsql.tree.SqlTreeNode; 6 | import org.slf4j.Logger; 7 | import org.slf4j.LoggerFactory; 8 | 9 | import java.lang.reflect.Constructor; 10 | import java.lang.reflect.InvocationTargetException; 11 | import java.util.HashMap; 12 | 13 | /** 14 | * Created by talus on 16/6/2. 15 | */ 16 | 17 | class OperatorBuilder { 18 | 19 | private static HashMap> operatorMap = new HashMap<>(); 20 | 21 | static { 22 | initOperatorMap(); 23 | } 24 | 25 | private static void initOperatorMap() { 26 | operatorMap.put(NodeType.CROWD_JOIN, CrowdJoinOperator.class); 27 | operatorMap.put(NodeType.CROWD_EQ, CrowdEQOperator.class); 28 | operatorMap.put(NodeType.CROWD_GT, CrowdGTOperator.class); 29 | operatorMap.put(NodeType.CROWD_LT, CrowdLTOperator.class); 30 | operatorMap.put(NodeType.CROWD_IN, CrowdInOperator.class); 31 | 32 | operatorMap.put(NodeType.JOIN, JoinOperator.class); 33 | operatorMap.put(NodeType.EQ, EQOperator.class); 34 | operatorMap.put(NodeType.GT, GTOperator.class); 35 | operatorMap.put(NodeType.LT, LTOperator.class); 36 | operatorMap.put(NodeType.PROJECT, ProjectOperator.class); 37 | 38 | operatorMap.put(NodeType.COLLECT, CollectOperator.class); 39 | operatorMap.put(NodeType.FILL, FillOperator.class); 40 | operatorMap.put(NodeType.SINGLELABEL, SingleLabel.class); 41 | operatorMap.put(NodeType.MULTILABEL, MultiLabel.class); 42 | 43 | operatorMap.put(NodeType.GRAPH_MODEL, CrowdGraphOperator.class); 44 | } 45 | 46 | public static Class createOperator(NodeType nodeType) { 47 | if (!operatorMap.containsKey(nodeType)) { 48 | return null; 49 | } 50 | 51 | return operatorMap.get(nodeType); 52 | } 53 | } 54 | 55 | public class QueryExecutor implements IQueryExecutor { 56 | 57 | private static String LOG_FORMAT = "##Query Executor##"; 58 | 59 | private static Logger LOG = LoggerFactory.getLogger(QueryExecutor.class); 60 | 61 | public BaseOperator buildOperator(SqlTreeNode node, String dbName) { 62 | Class operatorClass = OperatorBuilder.createOperator(node.getNodeType()); 63 | if (operatorClass == null) { 64 | LOG.error(String.format("no right operator, nodetype = %s", node.getNodeType())); 65 | return null; 66 | } 67 | try { 68 | Constructor cons = operatorClass.getConstructor(SqlTreeNode.class, String.class); 69 | BaseOperator operator = cons.newInstance(node, dbName); 70 | return operator; 71 | } catch (NoSuchMethodException | InstantiationException | IllegalAccessException | InvocationTargetException e) { 72 | LOG.error(String.format("can not create new instance for operator = %s", operatorClass.getName())); 73 | e.printStackTrace(); 74 | } 75 | return null; 76 | } 77 | 78 | public boolean execute(BaseOperator operator) { 79 | if (operator == null) return false; 80 | return operator.process(); 81 | } 82 | } 83 | -------------------------------------------------------------------------------- /CDB-Server/crowdexec/src/main/java/com/tsinghua/dbgroup/crowddb/crowdexec/query/RelationOperator.java: -------------------------------------------------------------------------------- 1 | package com.tsinghua.dbgroup.crowddb.crowdexec.query; 2 | 3 | import com.tsinghua.dbgroup.crowddb.crowdstorage.table.TableManager; 4 | import com.tsinghua.dbgroup.crowddb.crowdstorage.utils.TableHelper; 5 | import org.apache.commons.lang3.tuple.Pair; 6 | import org.slf4j.Logger; 7 | import org.slf4j.LoggerFactory; 8 | 9 | import java.sql.SQLException; 10 | import java.util.List; 11 | import java.util.Map; 12 | 13 | /** 14 | * Created by talus on 16/6/16. 15 | */ 16 | public class RelationOperator { 17 | 18 | private TableManager tm; 19 | 20 | private static String LOG_FORMAT = "##RelationOperator##"; 21 | 22 | private static Logger LOG = LoggerFactory.getLogger(RelationOperator.class); 23 | 24 | public RelationOperator() { 25 | tm = new TableManager(); 26 | } 27 | 28 | public void execEquals() { 29 | 30 | } 31 | 32 | public void execGreater() { 33 | 34 | } 35 | 36 | public void execLess() { 37 | 38 | } 39 | 40 | private List> extractColumns(String column) { 41 | Pair pair = TableHelper.unpackTableColumn(column); 42 | if (pair == null) { 43 | return null; 44 | } 45 | 46 | try { 47 | return tm.extractColumn(pair.getLeft(), pair.getRight(), null); 48 | } catch (SQLException e) { 49 | return null; 50 | } 51 | } 52 | } 53 | -------------------------------------------------------------------------------- /CDB-Server/crowdexec/src/main/java/com/tsinghua/dbgroup/crowddb/crowdexec/query/schema/HibernateSessionManager.java: -------------------------------------------------------------------------------- 1 | package com.tsinghua.dbgroup.crowddb.crowdexec.query.schema; 2 | 3 | /** 4 | * Created by talus on 16/6/19. 5 | */ 6 | 7 | import com.tsinghua.dbgroup.crowddb.crowdcore.configs.GlobalConfigs; 8 | import org.hibernate.Session; 9 | import org.hibernate.SessionFactory; 10 | import org.hibernate.cfg.Configuration; 11 | 12 | import java.util.Properties; 13 | 14 | public class HibernateSessionManager { 15 | 16 | private static SessionFactory sessionFactory; 17 | 18 | static 19 | { 20 | String username = GlobalConfigs.GlobalConfigs.getProperty("USERNAME"); 21 | String password = GlobalConfigs.GlobalConfigs.getProperty("PASSWORD"); 22 | String host = GlobalConfigs.GlobalConfigs.getProperty("HOST"); 23 | String port = GlobalConfigs.GlobalConfigs.getProperty("MYSQL_PORT"); 24 | String connUrl = String.format("jdbc:mysql://%s:%s/crowddb_meta", host, port); 25 | // String username = "crowddb"; 26 | // String password = "crowddb!password"; 27 | // String host = "166.111.71.172"; 28 | // String port = "3306"; 29 | // String connUrl = String.format("jdbc:mysql://%s:%s/crowddb_meta", host, port); 30 | 31 | // System.out.println(connUrl); 32 | Configuration configuration = new Configuration(); 33 | configuration.setProperty("hibernate.connection.url", connUrl); 34 | configuration.setProperty("hibernate.connection.username", username); 35 | configuration.setProperty("hibernate.connection.password", password); 36 | 37 | try 38 | { 39 | configuration.configure("hibernate/hibernate.cfg.xml"); 40 | sessionFactory = configuration.buildSessionFactory(); 41 | } 42 | catch(Exception ex) 43 | { 44 | ex.printStackTrace(); 45 | } 46 | } 47 | 48 | public static final ThreadLocal tl = new ThreadLocal(); 49 | 50 | public static Session currentSession() 51 | { 52 | // Session s = (Session)tl.get(); 53 | // if (s == null) 54 | // { 55 | // s = sessionFactory.openSession(); 56 | // tl.set(s); 57 | // } 58 | // 59 | // return s; 60 | return sessionFactory.getCurrentSession(); 61 | } 62 | 63 | public static void closeSession() 64 | { 65 | Session s = (Session)tl.get(); 66 | tl.set(null); 67 | if (s != null) 68 | { 69 | s.close(); 70 | } 71 | } 72 | } 73 | -------------------------------------------------------------------------------- /CDB-Server/crowdexec/src/main/resources/hibernate/hibernate.cfg.xml: -------------------------------------------------------------------------------- 1 | 2 | 5 | 6 | 7 | 8 | com.mysql.cj.jdbc.Driver 9 | org.hibernate.dialect.MySQLDialect 10 | 11 | 12 | 13 | false 14 | 50 15 | 25 16 | thread 17 | 18 | 19 | -------------------------------------------------------------------------------- /CDB-Server/crowdexec/src/main/resources/hibernate/query.hbm.xml: -------------------------------------------------------------------------------- 1 | 2 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | -------------------------------------------------------------------------------- /CDB-Server/crowdexec/src/test/java/com/tsinghua/dbgroup/crowddb/crowdexec/operator/CrowdEQOperatorTest.java: -------------------------------------------------------------------------------- 1 | package com.tsinghua.dbgroup.crowddb.crowdexec.operator; 2 | 3 | import org.junit.Test; 4 | 5 | import static org.junit.Assert.*; 6 | 7 | /** 8 | * Created by x-yu13 on 16/10/28. 9 | */ 10 | public class CrowdEQOperatorTest { 11 | @Test 12 | public void finish() throws Exception { 13 | 14 | } 15 | 16 | } -------------------------------------------------------------------------------- /CDB-Server/crowdexec/src/test/java/com/tsinghua/dbgroup/crowddb/crowdexec/query/schema/QueryControllerTest.java: -------------------------------------------------------------------------------- 1 | package com.tsinghua.dbgroup.crowddb.crowdexec.query.schema; 2 | 3 | import junit.framework.TestCase; 4 | import org.hibernate.Session; 5 | import org.hibernate.Transaction; 6 | 7 | /** 8 | * Created by talus on 16/6/19. 9 | */ 10 | public class QueryControllerTest extends TestCase { 11 | 12 | public void testQuery() { 13 | Session session = HibernateSessionManager.currentSession(); 14 | Query query = (Query)session.get(Query.class, new Integer(4)); 15 | assertNotNull(query); 16 | 17 | System.out.println("sql = " + query.sql); 18 | HibernateSessionManager.closeSession(); 19 | } 20 | 21 | public void testUpdate() { 22 | Session session = HibernateSessionManager.currentSession(); 23 | Transaction transaction = session.beginTransaction(); 24 | 25 | Query query = (Query)session.get(Query.class, new Integer(4)); 26 | assertNotNull(query); 27 | 28 | query.setSql("select * from user"); 29 | session.save(query); 30 | session.flush(); 31 | transaction.commit(); 32 | 33 | query = (Query)session.get(Query.class, new Integer(4)); 34 | System.out.println("sql = " + query.getSql()); 35 | 36 | HibernateSessionManager.closeSession(); 37 | } 38 | 39 | public void testInsert() { 40 | // Session session = HibernateSessionManager.currentSession(); 41 | // Transaction transaction = session.beginTransaction(); 42 | // Query query = new Query(); 43 | // query.setSql("select username from user"); 44 | // query.setTimestamp("2016-06-02 01:19:49"); 45 | // query.setUser(5); 46 | // query.setStatus("init"); 47 | // query.setCurrentSQLNodeId(1); 48 | // query.setResultTable("123"); 49 | // 50 | // session.save(query); 51 | // session.flush(); 52 | // transaction.commit(); 53 | // 54 | // query = (Query)session.get(Query.class, new Integer(4)); 55 | // System.out.println("sql = " + query.getSql()); 56 | // HibernateSessionManager.closeSession(); 57 | } 58 | 59 | public void testDelete() { 60 | 61 | } 62 | } -------------------------------------------------------------------------------- /CDB-Server/crowdplat/pom.xml: -------------------------------------------------------------------------------- 1 | 3 | 4 | crowddb 5 | com.tsinghua.dbgroup.crowddb 6 | 0.01 7 | 8 | 4.0.0 9 | 10 | crowdplat 11 | jar 12 | 13 | crowdplat 14 | http://maven.apache.org 15 | 16 | 17 | UTF-8 18 | 19 | 20 | 21 | 22 | junit 23 | junit 24 | 4.12 25 | test 26 | 27 | 28 | 29 | org.apache.httpcomponents 30 | httpclient 31 | 4.5.2 32 | 33 | 34 | 35 | org.apache.httpcomponents 36 | httpmime 37 | 4.5.2 38 | 39 | 40 | 41 | org.apache.commons 42 | commons-csv 43 | 1.4 44 | 45 | 46 | 47 | com.google.code.gson 48 | gson 49 | 2.8.0 50 | 51 | 52 | com.tsinghua.dbgroup.crowddb 53 | crowdcore 54 | 0.01 55 | 56 | 57 | 58 | -------------------------------------------------------------------------------- /CDB-Server/crowdplat/src/main/java/com/tsinghua/dbgroup/crowddb/crowdplat/TaskManager.java: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright (C) 2016-2016 by The Department of Computer Science and 3 | * Technology, Tsinghua University 4 | * 5 | * Redistribution of this file is permitted under the terms of 6 | * the BSD license. 7 | * 8 | * Author : XuepingWeng 9 | * Created : 10/8/16 9:14 PM 10 | * Modified : 11 | * Contact : wxping715@gmail.com 12 | */ 13 | 14 | package com.tsinghua.dbgroup.crowddb.crowdplat; 15 | 16 | import com.tsinghua.dbgroup.crowddb.crowdcore.configs.GlobalConfigs; 17 | import com.tsinghua.dbgroup.crowddb.crowdplat.core.Task; 18 | import com.tsinghua.dbgroup.crowddb.crowdplat.core.TaskCategory; 19 | import com.tsinghua.dbgroup.crowddb.crowdplat.core.TaskStatus; 20 | import com.tsinghua.dbgroup.crowddb.crowdplat.core.TaskType; 21 | import com.tsinghua.dbgroup.crowddb.crowdplat.engine.APEngine; 22 | import com.tsinghua.dbgroup.crowddb.crowdplat.result.BaseResult; 23 | import com.tsinghua.dbgroup.crowddb.crowdplat.schema.BaseSchema; 24 | import org.slf4j.Logger; 25 | import org.slf4j.LoggerFactory; 26 | 27 | import java.util.HashMap; 28 | import java.util.List; 29 | 30 | public class TaskManager { 31 | 32 | private static String LOG_FORMAT = "##TaskManager##"; 33 | 34 | private static Logger LOG = LoggerFactory.getLogger(TaskManager.class); 35 | 36 | private static boolean simulation = Boolean.valueOf(GlobalConfigs.GlobalConfigs.getProperty("SIMULATION")); 37 | 38 | public static Task createTask(TaskCategory taskCategory, TaskType taskType, List schemas) { 39 | Task task = new Task(schemas, taskCategory, taskType); 40 | return task; 41 | } 42 | 43 | public static boolean uploadTask(Task task) { 44 | if (simulation) return true; 45 | 46 | APEngine engine = new APEngine(); 47 | boolean result = engine.uploadTask(task); 48 | if (result) { 49 | LOG.info(String.format("upload task successfully, task_id = %s", task.getId())); 50 | } else { 51 | LOG.error(String.format("upload task failed, task_id = %s", task.getId())); 52 | } 53 | return result; 54 | // return true; 55 | } 56 | 57 | public static TaskStatus checkStatus(Task task) { 58 | if (simulation) return TaskStatus.FINISHED; 59 | 60 | APEngine engine = new APEngine(); 61 | TaskStatus taskStatus = engine.checkStatus(task.getId()); 62 | if (taskStatus == TaskStatus.FINISHED) { 63 | LOG.info(String.format("task = %s finished", task.getId())); 64 | } 65 | return taskStatus; 66 | } 67 | 68 | public static boolean pullRequests(Task task) { 69 | APEngine engine = new APEngine(); 70 | 71 | HashMap res = null; 72 | if (!simulation) { 73 | res = engine.pullResults(task); 74 | } else { 75 | res = task.mockResult(); 76 | } 77 | 78 | if (res != null) { 79 | // HashMap stringRes = task.packetResults(res); 80 | // task.extractResults(stringRes); 81 | task.extractResults(res); 82 | LOG.info(String.format("extract results successfully, task_id = %s", task.getId())); 83 | return true; 84 | } 85 | return false; 86 | } 87 | } 88 | -------------------------------------------------------------------------------- /CDB-Server/crowdplat/src/main/java/com/tsinghua/dbgroup/crowddb/crowdplat/core/Question.java: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright (C) 2016-2016 by The Department of Computer Science and 3 | * Technology, Tsinghua University 4 | * 5 | * Redistribution of this file is permitted under the terms of 6 | * the BSD license. 7 | * 8 | * Author : XuepingWeng 9 | * Created : 10/8/16 8:20 PM 10 | * Modified : 11 | * Contact : wxping715@gmail.com 12 | */ 13 | 14 | package com.tsinghua.dbgroup.crowddb.crowdplat.core; 15 | 16 | import com.google.gson.annotations.Expose; 17 | 18 | import java.util.List; 19 | 20 | public class Question { 21 | 22 | @Expose 23 | private String id; 24 | 25 | @Expose 26 | private String url; 27 | 28 | @Expose 29 | private String schemaId; 30 | 31 | @Expose 32 | private List columns; 33 | 34 | @Expose 35 | private String attribute; 36 | 37 | @Expose 38 | private int limit; 39 | 40 | @Expose 41 | private List options; 42 | 43 | @Expose 44 | private String content; 45 | 46 | public Question(String id, String content) { 47 | this.id = id; 48 | } 49 | 50 | public String getId() { 51 | return id; 52 | } 53 | 54 | public void setId(String id) { 55 | this.id = id; 56 | } 57 | 58 | public String getUrl() { 59 | return url; 60 | } 61 | 62 | public void setUrl(String url) { 63 | this.url = url; 64 | } 65 | 66 | public String getSchemaId() { 67 | return schemaId; 68 | } 69 | 70 | public void setSchemaId(String schemaId) { 71 | this.schemaId = schemaId; 72 | } 73 | 74 | public List getColumns() { 75 | return columns; 76 | } 77 | 78 | public void setColumns(List columns) { 79 | this.columns = columns; 80 | } 81 | 82 | public String getAttribute() { 83 | return attribute; 84 | } 85 | 86 | public void setAttribute(String attribute) { 87 | this.attribute = attribute; 88 | } 89 | 90 | public List getOptions() { 91 | return options; 92 | } 93 | 94 | public void setOptions(List options) { 95 | this.options = options; 96 | } 97 | 98 | public String getContent() { 99 | return content; 100 | } 101 | 102 | public void setContent(String content) { 103 | this.content = content; 104 | } 105 | 106 | public int getLimit() { 107 | return limit; 108 | } 109 | 110 | public void setLimit(int limit) { 111 | this.limit = limit; 112 | } 113 | } -------------------------------------------------------------------------------- /CDB-Server/crowdplat/src/main/java/com/tsinghua/dbgroup/crowddb/crowdplat/core/QuestionBuilder.java: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright (C) 2016-2016 by The Department of Computer Science and 3 | * Technology, Tsinghua University 4 | * 5 | * Redistribution of this file is permitted under the terms of 6 | * the BSD license. 7 | * 8 | * Author : XuepingWeng 9 | * Created : 11/8/16 4:23 PM 10 | * Modified : 11 | * Contact : wxping715@gmail.com 12 | */ 13 | 14 | package com.tsinghua.dbgroup.crowddb.crowdplat.core; 15 | 16 | import com.tsinghua.dbgroup.crowddb.crowdplat.schema.BaseSchema; 17 | import com.tsinghua.dbgroup.crowddb.crowdplat.schema.impl.*; 18 | 19 | public class QuestionBuilder { 20 | 21 | public Question createQuestion(TaskCategory taskCategory, TaskType taskType, String questionId, BaseSchema schema) { 22 | Question question = new Question(questionId, ""); 23 | question.setSchemaId(schema.getId()); 24 | String template = null; 25 | String content = null; 26 | 27 | switch (schema.getType()) { 28 | case COLLECT_SCHEMA: 29 | 30 | CollectionSchema collectionSchema = (CollectionSchema) schema; 31 | question.setAttribute(collectionSchema.getAttribute()); 32 | question.setLimit(collectionSchema.getLimit()); 33 | question.setColumns(collectionSchema.getColumns()); 34 | content = String.format(TaskConfigure.COLLECT_TEMPLATE, collectionSchema.getAttribute()); 35 | 36 | question.setContent(content); 37 | 38 | break; 39 | 40 | case JOIN_SCHEMA: 41 | JudgementSchema judgementSchema = (JudgementSchema) schema; 42 | content = String.format(TaskConfigure.JUDGEMENT_TEMPLATE, judgementSchema.getLeftKey(), 43 | TaskConfigure.COMPARISION_WORDS.get(taskType), judgementSchema.getRightKey()); 44 | question.setContent(content); 45 | 46 | break; 47 | 48 | case OPTION_SCHEMA: 49 | OptionsSchema optionsSchema = (OptionsSchema) schema; 50 | 51 | question.setOptions(optionsSchema.getOptions()); 52 | content = String.format(TaskConfigure.IN_TEMPLATE); 53 | question.setContent(content); 54 | 55 | break; 56 | 57 | case FILL_SCHEMA: 58 | FillSchema fillSchema = (FillSchema) schema; 59 | question.setColumns(fillSchema.getColumns()); 60 | question.setAttribute(fillSchema.getValue()); 61 | 62 | String columnStr = String.join(", ", fillSchema.getColumns()); 63 | content = String.format(TaskConfigure.FILL_TEMPLATE, columnStr, fillSchema.getValue()); 64 | question.setContent(content); 65 | 66 | break; 67 | 68 | case LABEL_SCHEMA: 69 | LabelSchema labelSchema = (LabelSchema) schema; 70 | question.setAttribute(labelSchema.getAttribute()); 71 | question.setOptions(labelSchema.getOptions()); 72 | question.setUrl(labelSchema.getUrl()); 73 | 74 | template = taskType == TaskType.SINGLE_LABEL ? TaskConfigure.SINGLE_LABEL_TEMPLATE : TaskConfigure.MULT_LABEL_TEMPLATE; 75 | content = String.format(template, labelSchema.getAttribute()); 76 | question.setContent(content); 77 | 78 | break; 79 | } 80 | return question; 81 | } 82 | } 83 | -------------------------------------------------------------------------------- /CDB-Server/crowdplat/src/main/java/com/tsinghua/dbgroup/crowddb/crowdplat/core/TaskCategory.java: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright (C) 2016-2016 by The Department of Computer Science and 3 | * Technology, Tsinghua University 4 | * 5 | * Redistribution of this file is permitted under the terms of 6 | * the BSD license. 7 | * 8 | * Author : XuepingWeng 9 | * Created : 10/12/16 2:50 PM 10 | * Modified : 11 | * Contact : wxping715@gmail.com 12 | */ 13 | 14 | package com.tsinghua.dbgroup.crowddb.crowdplat.core; 15 | 16 | public enum TaskCategory { 17 | TEXT("text"), IMAGE("image"), AUDIO("audio"); 18 | 19 | private final String category; 20 | 21 | private TaskCategory(final String category) { 22 | this.category = category; 23 | } 24 | 25 | @Override 26 | public String toString() { 27 | return this.category; 28 | } 29 | } -------------------------------------------------------------------------------- /CDB-Server/crowdplat/src/main/java/com/tsinghua/dbgroup/crowddb/crowdplat/core/TaskConfigure.java: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright (C) 2016-2016 by The Department of Computer Science and 3 | * Technology, Tsinghua University 4 | * 5 | * Redistribution of this file is permitted under the terms of 6 | * the BSD license. 7 | * 8 | * Author : XuepingWeng 9 | * Created : 10/8/16 4:25 PM 10 | * Modified : 11 | * Contact : wxping715@gmail.com 12 | */ 13 | 14 | 15 | package com.tsinghua.dbgroup.crowddb.crowdplat.core; 16 | 17 | import com.tsinghua.dbgroup.crowddb.crowdcore.configs.GlobalConfigs; 18 | 19 | import java.util.HashMap; 20 | 21 | 22 | public class TaskConfigure { 23 | 24 | /* 25 | * Request Urls 26 | * */ 27 | private static final String SERVER_HOST = GlobalConfigs.GlobalConfigs.getProperty("ASSIGNMENT_SERVER_URL"); 28 | public static final String UPLOAD_TASK_URL = SERVER_HOST + "/upload/"; 29 | public static final String CHECK_STATUS_URL = SERVER_HOST + "/check/"; 30 | public static final String DOWNLOAD_RESULTS_URL = SERVER_HOST + "/results/"; 31 | 32 | /* 33 | * Task Options 34 | * */ 35 | public static HashMap OPTIONS; 36 | 37 | /* 38 | * Task Templates 39 | * */ 40 | public static final String JUDGEMENT_TEMPLATE = "Please judge whether %s is %s %s ？"; 41 | public static final String COLLECT_TEMPLATE = "Please fill those blanks with appropriate tags for attribute %s"; 42 | public static final String IN_TEMPLATE = "Please choose options for this question"; 43 | public static final String COLLECT_IN_TEMPLATE = "Please choose best tags in candidates for attribute %s"; 44 | public static final String FILL_TEMPLATE = "Please fill the %s for entity %s"; 45 | public static final String SINGLE_LABEL_TEMPLATE = "Please tag single label for %s in this picture"; 46 | public static final String MULT_LABEL_TEMPLATE = "Please tag multiple labels for %s in this picture"; 47 | public static HashMap COMPARISION_WORDS; 48 | 49 | static { 50 | packetOptions(); 51 | initComparsionWords(); 52 | } 53 | 54 | private static void packetOptions() { 55 | OPTIONS = new HashMap<>(); 56 | OPTIONS.put("task_id", ""); 57 | OPTIONS.put("task_category", ""); 58 | OPTIONS.put("task_type", ""); 59 | OPTIONS.put("title", "Please compare those two entities cksjkkjsdfs"); 60 | OPTIONS.put("description", "Please compare those two entities"); 61 | OPTIONS.put("keywords", "Comparison"); 62 | OPTIONS.put("platform", (String) GlobalConfigs.GlobalConfigs.getOrDefault("DEFAULT_PLATFORM", "AMT")); 63 | OPTIONS.put("reward", (String) GlobalConfigs.GlobalConfigs.getOrDefault("MAX_ASSIGNMENT", "REWARD")); 64 | OPTIONS.put("duration", "120"); 65 | OPTIONS.put("approval_delay", "15"); 66 | OPTIONS.put("max_assignments", (String) GlobalConfigs.GlobalConfigs.getOrDefault("MAX_ASSIGNMENT", "5")); 67 | OPTIONS.put("lifetime", "15"); 68 | OPTIONS.put("q_type", "FREE"); 69 | } 70 | 71 | private static void initComparsionWords() { 72 | COMPARISION_WORDS = new HashMap<>(); 73 | COMPARISION_WORDS.put(TaskType.CROWD_EQUAL, "equal to"); 74 | COMPARISION_WORDS.put(TaskType.CROWD_NOT_EQUAL, "not equal to"); 75 | COMPARISION_WORDS.put(TaskType.CROWD_GT, "greater than"); 76 | COMPARISION_WORDS.put(TaskType.CROWD_GE, "greater than or equal to"); 77 | COMPARISION_WORDS.put(TaskType.CROWD_LT, "smaller than"); 78 | COMPARISION_WORDS.put(TaskType.CROWD_LE, "smaller than or equal to"); 79 | } 80 | } 81 | -------------------------------------------------------------------------------- /CDB-Server/crowdplat/src/main/java/com/tsinghua/dbgroup/crowddb/crowdplat/core/TaskPair.java: -------------------------------------------------------------------------------- 1 | package com.tsinghua.dbgroup.crowddb.crowdplat.core; 2 | 3 | /** 4 | * Created by x-yu13 on 16/10/28. 5 | */ 6 | public class TaskPair { 7 | String id1,id2,value1,value2; 8 | 9 | public TaskPair(String id1, String id2, String value1, String value2) { 10 | this.id1 = id1; 11 | this.id2 = id2; 12 | this.value1 = value1; 13 | this.value2 = value2; 14 | } 15 | 16 | public String getId1() { 17 | return id1; 18 | } 19 | 20 | public void setId1(String id1) { 21 | this.id1 = id1; 22 | } 23 | 24 | public String getId2() { 25 | return id2; 26 | } 27 | 28 | public void setId2(String id2) { 29 | this.id2 = id2; 30 | } 31 | 32 | public String getValue1() { 33 | return value1; 34 | } 35 | 36 | public void setValue1(String value1) { 37 | this.value1 = value1; 38 | } 39 | 40 | public String getValue2() { 41 | return value2; 42 | } 43 | 44 | public void setValue2(String value2) { 45 | this.value2 = value2; 46 | } 47 | } 48 | -------------------------------------------------------------------------------- /CDB-Server/crowdplat/src/main/java/com/tsinghua/dbgroup/crowddb/crowdplat/core/TaskStatus.java: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright (C) 2016-2016 by The Department of Computer Science and 3 | * Technology, Tsinghua University 4 | * 5 | * Redistribution of this file is permitted under the terms of 6 | * the BSD license. 7 | * 8 | * Author : XuepingWeng 9 | * Created : 10/8/16 9:00 PM 10 | * Modified : 11 | * Contact : wxping715@gmail.com 12 | */ 13 | 14 | package com.tsinghua.dbgroup.crowddb.crowdplat.core; 15 | 16 | public enum TaskStatus { 17 | FINISHING, 18 | FINISHED, 19 | } 20 | -------------------------------------------------------------------------------- /CDB-Server/crowdplat/src/main/java/com/tsinghua/dbgroup/crowddb/crowdplat/core/TaskType.java: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright (C) 2016-2016 by The Department of Computer Science and 3 | * Technology, Tsinghua University 4 | * 5 | * Redistribution of this file is permitted under the terms of 6 | * the BSD license. 7 | * 8 | * Author : XuepingWeng 9 | * Created : 10/8/16 9:01 PM 10 | * Modified : 11 | * Contact : wxping715@gmail.com 12 | */ 13 | 14 | package com.tsinghua.dbgroup.crowddb.crowdplat.core; 15 | 16 | public enum TaskType { 17 | 18 | /** 19 | * Judgement Questions 20 | * */ 21 | CROWD_EQUAL("crowd_eq"), CROWD_NOT_EQUAL("crowd_not_eq"), CROWD_GT("crowd_gt"), 22 | CROWD_GE("crowd_ge"), CROWD_LT("crowd_lt"), CROWD_LE("crowd_le"),CROWD_JOIN("crowd_join"), 23 | 24 | /** 25 | * Multiple Options Questions 26 | * */ 27 | CROWD_IN("crowd_in"), 28 | 29 | /** 30 | * Collection Questions 31 | * */ 32 | COLLECT("collect"), 33 | 34 | /** 35 | * Label Questions 36 | */ 37 | SINGLE_LABEL("single_label"), MULTI_LABEL("multi_label"), 38 | 39 | /** 40 | * FILL Questions 41 | */ 42 | FILL("fill") 43 | 44 | ; 45 | 46 | private String type; 47 | 48 | private TaskType(String type) { 49 | this.type = type; 50 | } 51 | 52 | @Override 53 | public String toString() { 54 | return this.type; 55 | } 56 | } -------------------------------------------------------------------------------- /CDB-Server/crowdplat/src/main/java/com/tsinghua/dbgroup/crowddb/crowdplat/engine/ChinaCrowdsEngine.java: -------------------------------------------------------------------------------- 1 | package com.tsinghua.dbgroup.crowddb.crowdplat.engine; 2 | import com.tsinghua.dbgroup.crowddb.crowdplat.core.Question; 3 | import com.tsinghua.dbgroup.crowddb.crowdplat.core.Task; 4 | import com.tsinghua.dbgroup.crowddb.crowdplat.core.TaskStatus; 5 | import com.tsinghua.dbgroup.crowddb.crowdplat.result.BaseResult; 6 | 7 | import java.util.HashMap; 8 | import java.util.List; 9 | 10 | /** 11 | * Created by talus on 16/6/2. 12 | */ 13 | public class ChinaCrowdsEngine implements ICrowdEngine{ 14 | 15 | @Override 16 | public boolean uploadTask(Task task) { 17 | return false; 18 | } 19 | 20 | @Override 21 | public TaskStatus checkStatus(String taskId) { 22 | return null; 23 | } 24 | 25 | @Override 26 | public boolean appendData(String taskId, List questions) { 27 | return false; 28 | } 29 | 30 | @Override 31 | public HashMap pullResults(Task task) { 32 | return null; 33 | } 34 | } 35 | -------------------------------------------------------------------------------- /CDB-Server/crowdplat/src/main/java/com/tsinghua/dbgroup/crowddb/crowdplat/engine/ICrowdEngine.java: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright (C) 2016-2016 by The Department of Computer Science and 3 | * Technology, Tsinghua University 4 | * 5 | * Redistribution of this file is permitted under the terms of 6 | * the BSD license. 7 | * 8 | * Author : XuepingWeng 9 | * Created : 10/8/16 8:27 PM 10 | * Modified : 11 | * Contact : wxping715@gmail.com 12 | */ 13 | 14 | package com.tsinghua.dbgroup.crowddb.crowdplat.engine; 15 | import com.tsinghua.dbgroup.crowddb.crowdplat.core.Question; 16 | import com.tsinghua.dbgroup.crowddb.crowdplat.core.Task; 17 | import com.tsinghua.dbgroup.crowddb.crowdplat.core.TaskStatus; 18 | import com.tsinghua.dbgroup.crowddb.crowdplat.result.BaseResult; 19 | 20 | import java.util.HashMap; 21 | import java.util.List; 22 | 23 | /** 24 | * Created by talus on 16/6/2. 25 | */ 26 | 27 | public interface ICrowdEngine { 28 | 29 | public boolean uploadTask(Task task); 30 | 31 | public TaskStatus checkStatus(String taskId); 32 | 33 | public boolean appendData(String taskId, List questions); 34 | 35 | public HashMap pullResults(Task task); 36 | } 37 | -------------------------------------------------------------------------------- /CDB-Server/crowdplat/src/main/java/com/tsinghua/dbgroup/crowddb/crowdplat/http/IHttpRequest.java: -------------------------------------------------------------------------------- 1 | package com.tsinghua.dbgroup.crowddb.crowdplat.http; 2 | 3 | import com.google.gson.JsonObject; 4 | 5 | import java.util.Map; 6 | 7 | /** 8 | * Created by talus on 16/6/1. 9 | */ 10 | public interface IHttpRequest{ 11 | 12 | public JsonObject sendGetRequest(String url, Map params); 13 | 14 | public JsonObject sendPostRequest(String url, Map getParams, Map postParams); 15 | 16 | public JsonObject sendPostRequest(String url, Map getParams, Map postParams, Map files); 17 | } 18 | -------------------------------------------------------------------------------- /CDB-Server/crowdplat/src/main/java/com/tsinghua/dbgroup/crowddb/crowdplat/result/BaseResult.java: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright (C) 2016-2016 by The Department of Computer Science and 3 | * Technology, Tsinghua University 4 | * 5 | * Redistribution of this file is permitted under the terms of 6 | * the BSD license. 7 | * 8 | * Author : XuepingWeng 9 | * Created : 11/8/16 4:56 PM 10 | * Modified : 11 | * Contact : wxping715@gmail.com 12 | */ 13 | 14 | package com.tsinghua.dbgroup.crowddb.crowdplat.result; 15 | 16 | public abstract class BaseResult { 17 | 18 | private String id; 19 | 20 | public BaseResult(String id) { 21 | this.id = id; 22 | } 23 | 24 | public String getId() { 25 | return id; 26 | } 27 | 28 | public void setId(String id) { 29 | this.id = id; 30 | } 31 | } 32 | -------------------------------------------------------------------------------- /CDB-Server/crowdplat/src/main/java/com/tsinghua/dbgroup/crowddb/crowdplat/result/impl/CollectionResult.java: -------------------------------------------------------------------------------- 1 | package com.tsinghua.dbgroup.crowddb.crowdplat.result.impl; 2 | 3 | import com.tsinghua.dbgroup.crowddb.crowdplat.result.BaseResult; 4 | 5 | import java.util.HashMap; 6 | import java.util.List; 7 | 8 | /** 9 | * Created by talus on 11/30/16. 10 | */ 11 | public class CollectionResult extends BaseResult { 12 | 13 | private List> answer; 14 | 15 | public CollectionResult(String id, List> answer) { 16 | super(id); 17 | this.setAnswer(answer); 18 | } 19 | 20 | public List> getAnswer() { 21 | return answer; 22 | } 23 | 24 | public void setAnswer(List> answer) { 25 | this.answer = answer; 26 | } 27 | } 28 | -------------------------------------------------------------------------------- /CDB-Server/crowdplat/src/main/java/com/tsinghua/dbgroup/crowddb/crowdplat/result/impl/ColumnsResult.java: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright (C) 2016-2016 by The Department of Computer Science and 3 | * Technology, Tsinghua University 4 | * 5 | * Redistribution of this file is permitted under the terms of 6 | * the BSD license. 7 | * 8 | * Author : XuepingWeng 9 | * Created : 11/8/16 4:58 PM 10 | * Modified : 11 | * Contact : wxping715@gmail.com 12 | */ 13 | 14 | package com.tsinghua.dbgroup.crowddb.crowdplat.result.impl; 15 | 16 | import com.tsinghua.dbgroup.crowddb.crowdplat.result.BaseResult; 17 | 18 | import java.util.HashMap; 19 | import java.util.List; 20 | 21 | public class ColumnsResult extends BaseResult { 22 | 23 | private HashMap answer; 24 | 25 | public ColumnsResult(String id, HashMap answer) { 26 | super(id); 27 | setAnswer(answer); 28 | } 29 | 30 | public HashMap getAnswer() { 31 | return answer; 32 | } 33 | 34 | public void setAnswer(HashMap answer) { 35 | this.answer = answer; 36 | } 37 | 38 | @Override 39 | public String toString() { 40 | return "ColumnsResult{" + 41 | "id=" + getId() + 42 | ", answer=" + answer + 43 | '}'; 44 | } 45 | } 46 | -------------------------------------------------------------------------------- /CDB-Server/crowdplat/src/main/java/com/tsinghua/dbgroup/crowddb/crowdplat/result/impl/JudgementResult.java: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright (C) 2016-2016 by The Department of Computer Science and 3 | * Technology, Tsinghua University 4 | * 5 | * Redistribution of this file is permitted under the terms of 6 | * the BSD license. 7 | * 8 | * Author : XuepingWeng 9 | * Created : 11/8/16 4:58 PM 10 | * Modified : 11 | * Contact : wxping715@gmail.com 12 | */ 13 | 14 | package com.tsinghua.dbgroup.crowddb.crowdplat.result.impl; 15 | 16 | import com.tsinghua.dbgroup.crowddb.crowdplat.result.BaseResult; 17 | 18 | public class JudgementResult extends BaseResult { 19 | 20 | private Integer answer; 21 | 22 | public JudgementResult(String id, int value) { 23 | super(id); 24 | this.setAnswer(value); 25 | } 26 | 27 | public Integer getAnswer() { 28 | return answer; 29 | } 30 | 31 | public void setAnswer(Integer answer) { 32 | this.answer = answer; 33 | } 34 | 35 | @Override 36 | public String toString() { 37 | return "JudgementResult{" + 38 | "id=" + getId() + 39 | ", answer=" + answer + 40 | '}'; 41 | } 42 | } 43 | -------------------------------------------------------------------------------- /CDB-Server/crowdplat/src/main/java/com/tsinghua/dbgroup/crowddb/crowdplat/result/impl/OptionsResult.java: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright (C) 2016-2016 by The Department of Computer Science and 3 | * Technology, Tsinghua University 4 | * 5 | * Redistribution of this file is permitted under the terms of 6 | * the BSD license. 7 | * 8 | * Author : XuepingWeng 9 | * Created : 11/8/16 4:57 PM 10 | * Modified : 11 | * Contact : wxping715@gmail.com 12 | */ 13 | 14 | package com.tsinghua.dbgroup.crowddb.crowdplat.result.impl; 15 | 16 | import com.tsinghua.dbgroup.crowddb.crowdplat.result.BaseResult; 17 | 18 | import java.util.List; 19 | 20 | public class OptionsResult extends BaseResult { 21 | 22 | private List answer; 23 | 24 | public OptionsResult(String id, List options) { 25 | super(id); 26 | this.setAnswer(options); 27 | } 28 | 29 | public List getAnswer() { 30 | return answer; 31 | } 32 | 33 | public void setAnswer(List answer) { 34 | this.answer = answer; 35 | } 36 | 37 | @Override 38 | public String toString() { 39 | return "OptionsResult{" + 40 | "id=" + getId() + 41 | ", answer=" + answer + 42 | '}'; 43 | } 44 | } 45 | -------------------------------------------------------------------------------- /CDB-Server/crowdplat/src/main/java/com/tsinghua/dbgroup/crowddb/crowdplat/schema/BaseSchema.java: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright (C) 2016-2016 by The Department of Computer Science and 3 | * Technology, Tsinghua University 4 | * 5 | * Redistribution of this file is permitted under the terms of 6 | * the BSD license. 7 | * 8 | * Author : XuepingWeng 9 | * Created : 11/8/16 4:05 PM 10 | * Modified : 11 | * Contact : wxping715@gmail.com 12 | */ 13 | 14 | package com.tsinghua.dbgroup.crowddb.crowdplat.schema; 15 | 16 | public abstract class BaseSchema implements ISchema { 17 | 18 | private String id; 19 | 20 | public BaseSchema(String id) { 21 | this.id = id; 22 | } 23 | 24 | public String getId() { 25 | return id; 26 | } 27 | 28 | public void setId(String id) { 29 | this.id = id; 30 | } 31 | 32 | @Override 33 | public SchemaType getType() { 34 | return SchemaType.UNKNOWN; 35 | } 36 | } 37 | -------------------------------------------------------------------------------- /CDB-Server/crowdplat/src/main/java/com/tsinghua/dbgroup/crowddb/crowdplat/schema/ISchema.java: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright (C) 2016-2016 by The Department of Computer Science and 3 | * Technology, Tsinghua University 4 | * 5 | * Redistribution of this file is permitted under the terms of 6 | * the BSD license. 7 | * 8 | * Author : XuepingWeng 9 | * Created : 11/8/16 4:13 PM 10 | * Modified : 11 | * Contact : wxping715@gmail.com 12 | */ 13 | 14 | package com.tsinghua.dbgroup.crowddb.crowdplat.schema; 15 | 16 | public interface ISchema { 17 | 18 | public SchemaType getType(); 19 | 20 | } 21 | -------------------------------------------------------------------------------- /CDB-Server/crowdplat/src/main/java/com/tsinghua/dbgroup/crowddb/crowdplat/schema/SchemaType.java: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright (C) 2016-2016 by The Department of Computer Science and 3 | * Technology, Tsinghua University 4 | * 5 | * Redistribution of this file is permitted under the terms of 6 | * the BSD license. 7 | * 8 | * Author : XuepingWeng 9 | * Created : 11/8/16 4:14 PM 10 | * Modified : 11 | * Contact : wxping715@gmail.com 12 | */ 13 | 14 | package com.tsinghua.dbgroup.crowddb.crowdplat.schema; 15 | 16 | public enum SchemaType { 17 | JOIN_SCHEMA("join_schema"), OPTION_SCHEMA("in_schema"), COLLECT_SCHEMA("collect_schema"), 18 | FILL_SCHEMA("fill_schema"), LABEL_SCHEMA("label_schema"), UNKNOWN("unknown"); 19 | 20 | private String type; 21 | 22 | private SchemaType(String type) { 23 | this.type = type; 24 | } 25 | 26 | @Override 27 | public String toString() { 28 | return this.type; 29 | } 30 | } 31 | -------------------------------------------------------------------------------- /CDB-Server/crowdplat/src/main/java/com/tsinghua/dbgroup/crowddb/crowdplat/schema/impl/CollectionSchema.java: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright (C) 2016-2016 by The Department of Computer Science and 3 | * Technology, Tsinghua University 4 | * 5 | * Redistribution of this file is permitted under the terms of 6 | * the BSD license. 7 | * 8 | * Author : XuepingWeng 9 | * Created : 11/8/16 4:55 PM 10 | * Modified : 11 | * Contact : wxping715@gmail.com 12 | */ 13 | 14 | package com.tsinghua.dbgroup.crowddb.crowdplat.schema.impl; 15 | import com.tsinghua.dbgroup.crowddb.crowdplat.schema.BaseSchema; 16 | import com.tsinghua.dbgroup.crowddb.crowdplat.schema.ISchema; 17 | import com.tsinghua.dbgroup.crowddb.crowdplat.schema.SchemaType; 18 | 19 | import java.util.List; 20 | 21 | public class CollectionSchema extends BaseSchema implements ISchema { 22 | 23 | private List columns; 24 | 25 | private String attribute; 26 | 27 | private int limit; 28 | 29 | public CollectionSchema(String id, String attribute, int limit, List columns) { 30 | super(id); 31 | setAttribute(attribute); 32 | setLimit(limit); 33 | setColumns(columns); 34 | } 35 | 36 | public int getLimit() { 37 | return limit; 38 | } 39 | 40 | public void setLimit(int limit) { 41 | this.limit = limit; 42 | } 43 | 44 | public String getAttribute() { 45 | return attribute; 46 | } 47 | 48 | public void setAttribute(String attribute) { 49 | this.attribute = attribute; 50 | } 51 | 52 | public List getColumns() { 53 | return columns; 54 | } 55 | 56 | public void setColumns(List columns) { 57 | this.columns = columns; 58 | } 59 | 60 | @Override 61 | public SchemaType getType() { 62 | return SchemaType.COLLECT_SCHEMA; 63 | } 64 | 65 | @Override 66 | public String toString() { 67 | return "CollectionSchema{" + 68 | "limit='" + limit + '\'' + 69 | ", attribute='" + attribute + '\'' + 70 | ", c='" + columns + '\'' + 71 | '}'; 72 | } 73 | } 74 | -------------------------------------------------------------------------------- /CDB-Server/crowdplat/src/main/java/com/tsinghua/dbgroup/crowddb/crowdplat/schema/impl/FillSchema.java: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright (C) 2016-2016 by The Department of Computer Science and 3 | * Technology, Tsinghua University 4 | * 5 | * Redistribution of this file is permitted under the terms of 6 | * the BSD license. 7 | * 8 | * Author : XuepingWeng 9 | * Created : 11/15/16 1:33 PM 10 | * Modified : 11 | * Contact : wxping715@gmail.com 12 | */ 13 | 14 | package com.tsinghua.dbgroup.crowddb.crowdplat.schema.impl; 15 | 16 | import com.tsinghua.dbgroup.crowddb.crowdplat.schema.BaseSchema; 17 | import com.tsinghua.dbgroup.crowddb.crowdplat.schema.SchemaType; 18 | 19 | import java.util.List; 20 | 21 | public class FillSchema extends BaseSchema { 22 | 23 | private List columns; 24 | 25 | private String value; 26 | 27 | public FillSchema(String id, String value, List columns) { 28 | super(id); 29 | setColumns(columns); 30 | setValue(value); 31 | } 32 | 33 | public List getColumns() { 34 | return columns; 35 | } 36 | 37 | public void setColumns(List columns) { 38 | this.columns = columns; 39 | } 40 | 41 | public String getValue() { 42 | return value; 43 | } 44 | 45 | public void setValue(String value) { 46 | this.value = value; 47 | } 48 | 49 | @Override 50 | public SchemaType getType() { 51 | return SchemaType.FILL_SCHEMA; 52 | } 53 | } 54 | -------------------------------------------------------------------------------- /CDB-Server/crowdplat/src/main/java/com/tsinghua/dbgroup/crowddb/crowdplat/schema/impl/JudgementSchema.java: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright (C) 2016-2016 by The Department of Computer Science and 3 | * Technology, Tsinghua University 4 | * 5 | * Redistribution of this file is permitted under the terms of 6 | * the BSD license. 7 | * 8 | * Author : XuepingWeng 9 | * Created : 11/8/16 4:55 PM 10 | * Modified : 11 | * Contact : wxping715@gmail.com 12 | */ 13 | 14 | package com.tsinghua.dbgroup.crowddb.crowdplat.schema.impl; 15 | 16 | import com.tsinghua.dbgroup.crowddb.crowdplat.schema.BaseSchema; 17 | import com.tsinghua.dbgroup.crowddb.crowdplat.schema.ISchema; 18 | import com.tsinghua.dbgroup.crowddb.crowdplat.schema.SchemaType; 19 | 20 | public class JudgementSchema extends BaseSchema implements ISchema { 21 | 22 | public static final String TYPE = "join_schema"; 23 | 24 | private String leftKey; 25 | 26 | private String rightKey; 27 | 28 | public JudgementSchema(String id, String leftKey, String rightKey) { 29 | super(id); 30 | this.leftKey = leftKey; 31 | this.rightKey = rightKey; 32 | } 33 | 34 | public String getLeftKey() { 35 | return leftKey; 36 | } 37 | 38 | public void setLeftKey(String leftKey) { 39 | this.leftKey = leftKey; 40 | } 41 | 42 | public String getRightKey() { 43 | return rightKey; 44 | } 45 | 46 | public void setRightKey(String rightKey) { 47 | this.rightKey = rightKey; 48 | } 49 | 50 | @Override 51 | public SchemaType getType() { 52 | return SchemaType.JOIN_SCHEMA; 53 | } 54 | 55 | @Override 56 | public String toString() { 57 | return "JudgementSchema{" + 58 | "leftKey='" + leftKey + '\'' + 59 | ", rightKey='" + rightKey + '\'' + 60 | '}'; 61 | } 62 | } 63 | -------------------------------------------------------------------------------- /CDB-Server/crowdplat/src/main/java/com/tsinghua/dbgroup/crowddb/crowdplat/schema/impl/LabelSchema.java: -------------------------------------------------------------------------------- 1 | package com.tsinghua.dbgroup.crowddb.crowdplat.schema.impl; 2 | 3 | import com.tsinghua.dbgroup.crowddb.crowdplat.schema.BaseSchema; 4 | import com.tsinghua.dbgroup.crowddb.crowdplat.schema.SchemaType; 5 | 6 | import java.util.List; 7 | 8 | /** 9 | * Created by talus on 11/17/16. 10 | */ 11 | public class LabelSchema extends BaseSchema{ 12 | 13 | private String url; 14 | 15 | private List options; 16 | 17 | private String attribute; 18 | 19 | public LabelSchema(String id, String attribute, String url, List options) { 20 | super(id); 21 | this.url = url; 22 | this.options = options; 23 | this.attribute = attribute; 24 | } 25 | 26 | public String getUrl() { 27 | return url; 28 | } 29 | 30 | public void setUrl(String url) { 31 | this.url = url; 32 | } 33 | 34 | public List getOptions() { 35 | return options; 36 | } 37 | 38 | public void setOptions(List options) { 39 | this.options = options; 40 | } 41 | 42 | public String getAttribute() { 43 | return attribute; 44 | } 45 | 46 | public void setAttribute(String attribute) { 47 | this.attribute = attribute; 48 | } 49 | 50 | @Override 51 | public SchemaType getType() { 52 | return SchemaType.LABEL_SCHEMA; 53 | } 54 | } 55 | -------------------------------------------------------------------------------- /CDB-Server/crowdplat/src/main/java/com/tsinghua/dbgroup/crowddb/crowdplat/schema/impl/OptionsSchema.java: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright (C) 2016-2016 by The Department of Computer Science and 3 | * Technology, Tsinghua University 4 | * 5 | * Redistribution of this file is permitted under the terms of 6 | * the BSD license. 7 | * 8 | * Author : XuepingWeng 9 | * Created : 11/8/16 4:55 PM 10 | * Modified : 11 | * Contact : wxping715@gmail.com 12 | */ 13 | 14 | package com.tsinghua.dbgroup.crowddb.crowdplat.schema.impl; 15 | 16 | import com.tsinghua.dbgroup.crowddb.crowdplat.schema.BaseSchema; 17 | import com.tsinghua.dbgroup.crowddb.crowdplat.schema.ISchema; 18 | import com.tsinghua.dbgroup.crowddb.crowdplat.schema.SchemaType; 19 | 20 | import java.util.List; 21 | 22 | public class OptionsSchema extends BaseSchema implements ISchema { 23 | 24 | private String keywords; 25 | 26 | private List options; 27 | 28 | public OptionsSchema(String id, String keywords, List options) { 29 | super(id); 30 | this.keywords = keywords; 31 | this.options = options; 32 | } 33 | 34 | public String getKeywords() { 35 | return keywords; 36 | } 37 | 38 | public void setKeywords(String keywords) { 39 | this.keywords = keywords; 40 | } 41 | 42 | public List getOptions() { 43 | return options; 44 | } 45 | 46 | public void setOptions(List options) { 47 | this.options = options; 48 | } 49 | 50 | @Override 51 | public SchemaType getType() { 52 | return SchemaType.OPTION_SCHEMA; 53 | } 54 | 55 | @Override 56 | public String toString() { 57 | return "OptionsSchema{" + 58 | "keywords='" + keywords + '\'' + 59 | ", options=" + options + 60 | '}'; 61 | } 62 | } 63 | -------------------------------------------------------------------------------- /CDB-Server/crowdplat/src/test/java/com/tsinghua/dbgroup/crowddb/crowdplat/engine/APEngineTest.java: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright (C) 2016-2016 by The Department of Computer Science and 3 | * Technology, Tsinghua University 4 | * 5 | * Redistribution of this file is permitted under the terms of 6 | * the BSD license. 7 | * 8 | * Author : XuepingWeng 9 | * Created : 11/9/16 7:58 PM 10 | * Modified : 11 | * Contact : wxping715@gmail.com 12 | */ 13 | 14 | package com.tsinghua.dbgroup.crowddb.crowdplat.engine; 15 | 16 | import com.tsinghua.dbgroup.crowddb.crowdplat.core.Task; 17 | import com.tsinghua.dbgroup.crowddb.crowdplat.core.TaskCategory; 18 | import com.tsinghua.dbgroup.crowddb.crowdplat.core.TaskType; 19 | import com.tsinghua.dbgroup.crowddb.crowdplat.result.BaseResult; 20 | import com.tsinghua.dbgroup.crowddb.crowdplat.result.impl.JudgementResult; 21 | import com.tsinghua.dbgroup.crowddb.crowdplat.result.impl.OptionsResult; 22 | import com.tsinghua.dbgroup.crowddb.crowdplat.schema.impl.JudgementSchema; 23 | import junit.framework.TestCase; 24 | import org.junit.Test; 25 | 26 | import java.util.ArrayList; 27 | import java.util.HashMap; 28 | import java.util.List; 29 | 30 | public class APEngineTest extends TestCase{ 31 | 32 | @Test 33 | public void testConvertResult() throws Exception { 34 | // APEngine engine = new APEngine(); 35 | // 36 | // List schemas = new ArrayList(); 37 | // 38 | // Task task = new Task(schemas, TaskCategory.TEXT, TaskType.CROWD_EQUAL); 39 | // String jsonString = "{\"q1\": {\"answer\": 1, \"id\": \"q1\"}}"; 40 | // 41 | // HashMap map = new HashMap<>(); 42 | // map = engine.(task, jsonString); 43 | // 44 | // for (HashMap.Entry entry: map.entrySet()) { 45 | // System.out.println(entry.getValue().toString()); 46 | // assertEquals(((JudgementResult)entry.getValue()).getAnswer(), new Integer(1)); 47 | // } 48 | // 49 | // jsonString = "{\"q1\": {\"answer\": [\"red\", \"green\", \"blue\"], \"id\": \"q1\"}}"; 50 | // task = new Task(schemas, TaskCategory.TEXT, TaskType.CROWD_IN); 51 | // map = engine.convertResult(task, jsonString); 52 | // List list = new ArrayList<>(); 53 | // list.add("red"); 54 | // list.add("green"); 55 | // list.add("blue"); 56 | // for (HashMap.Entry entry: map.entrySet()) { 57 | // OptionsResult optionsResult = (OptionsResult) entry.getValue(); 58 | // System.out.println(optionsResult.toString()); 59 | // assertEquals(optionsResult.getAnswer(), list); 60 | // } 61 | } 62 | } -------------------------------------------------------------------------------- /CDB-Server/crowdplat/src/test/java/com/tsinghua/dbgroup/crowddb/crowdplat/http/HttpRequestTest.java: -------------------------------------------------------------------------------- 1 | package com.tsinghua.dbgroup.crowddb.crowdplat.http; 2 | 3 | import com.google.gson.JsonObject; 4 | import junit.framework.TestCase; 5 | 6 | import java.util.HashMap; 7 | import java.util.Map; 8 | 9 | /** 10 | * Created by talus on 16/6/2. 11 | */ 12 | public class HttpRequestTest extends TestCase { 13 | 14 | private static String url = "http://www.chinacrowds.com:6789/account/login"; 15 | 16 | private HttpRequest request = null; 17 | 18 | public void testGetRequest() { 19 | JsonObject json = request.sendGetRequest(url, null); 20 | 21 | assertNotNull(json); 22 | assertEquals(json.get("code").getAsInt(), 1100); 23 | } 24 | 25 | public void testPostRequest() { 26 | Map params = new HashMap(); 27 | params.put("username", "talus"); 28 | params.put("password", "chinacrowds"); 29 | params.put("type", "worker"); 30 | 31 | JsonObject json = request.sendPostRequest(url, null, params); 32 | assertNotNull(json); 33 | assertEquals(json.get("code").getAsInt(), 0); 34 | } 35 | 36 | public void testPostRequestWithFile() { 37 | 38 | } 39 | 40 | @Override 41 | protected void setUp() throws Exception { 42 | request = new HttpRequest(); 43 | } 44 | 45 | @Override 46 | protected void tearDown() throws Exception { 47 | super.tearDown(); 48 | } 49 | } 50 | -------------------------------------------------------------------------------- /CDB-Server/crowdscheduler/pom.xml: -------------------------------------------------------------------------------- 1 | 3 | 4 | crowddb 5 | com.tsinghua.dbgroup.crowddb 6 | 0.01 7 | 8 | 4.0.0 9 | 10 | com.tsinghua.dbgroup.crowddb 11 | crowdscheduler 12 | jar 13 | 14 | crowdscheduler 15 | http://maven.apache.org 16 | 17 | 18 | UTF-8 19 | 20 | 21 | 22 | 23 | junit 24 | junit 25 | 4.12 26 | test 27 | 28 | 29 | com.tsinghua.dbgroup.crowddb 30 | crowdcore 31 | 0.01 32 | 33 | 34 | com.tsinghua.dbgroup.crowddb 35 | crowdsql 36 | 0.01 37 | 38 | 39 | com.tsinghua.dbgroup.crowddb 40 | crowdexec 41 | 0.01 42 | 43 | 44 | com.tsinghua.dbgroup.crowddb 45 | crowdplat 46 | 0.01 47 | 48 | 49 | 50 | com.jcraft 51 | jsch 52 | 0.1.54 53 | 54 | 55 | 56 | -------------------------------------------------------------------------------- /CDB-Server/crowdscheduler/src/main/java/com/tsinghua/dbgroup/crowddb/scheduler/Configs.java: -------------------------------------------------------------------------------- 1 | package com.tsinghua.dbgroup.crowddb.scheduler; 2 | import jdk.nashorn.internal.runtime.regexp.joni.Config; 3 | 4 | import java.io.FileInputStream; 5 | import java.io.IOException; 6 | import java.io.InputStream; 7 | import java.nio.file.Path; 8 | import java.util.Properties; 9 | 10 | /** 11 | * Created by talus on 16/6/20. 12 | */ 13 | public class Configs { 14 | 15 | public static Properties CONFIGS; 16 | 17 | static { 18 | try { 19 | CONFIGS = loadConfig(); 20 | } catch (IOException e) { 21 | CONFIGS = null; 22 | e.printStackTrace(); 23 | } 24 | } 25 | 26 | public static Properties loadConfig() throws IOException{ 27 | Properties prop = new Properties(); 28 | InputStream input = null; 29 | String configName = "resources/runtime.configs.properties"; 30 | 31 | String configPath = String.format("%s/%s", System.getProperty("user.dir"), configName); 32 | input = new FileInputStream(configPath); 33 | if (input == null) { 34 | throw new IOException("cann't find the config file " + configName); 35 | } 36 | prop.load(input); 37 | return prop; 38 | } 39 | } 40 | -------------------------------------------------------------------------------- /CDB-Server/crowdscheduler/src/main/java/com/tsinghua/dbgroup/crowddb/scheduler/Server.java: -------------------------------------------------------------------------------- 1 | package com.tsinghua.dbgroup.crowddb.scheduler; 2 | import com.tsinghua.dbgroup.crowddb.crowdcore.configs.GlobalConfigs; 3 | import com.tsinghua.dbgroup.crowddb.scheduler.dispatcher.QueryScheduler; 4 | import com.tsinghua.dbgroup.crowddb.scheduler.server.SocketServer; 5 | import org.slf4j.Logger; 6 | import org.slf4j.LoggerFactory; 7 | 8 | import java.text.DateFormat; 9 | import java.text.SimpleDateFormat; 10 | import java.util.Date; 11 | import java.util.Properties; 12 | import java.util.concurrent.Executors; 13 | import java.util.concurrent.ScheduledExecutorService; 14 | import java.util.concurrent.TimeUnit; 15 | 16 | /** 17 | * Created by talus on 16/6/20. 18 | */ 19 | public class Server { 20 | 21 | private static String LOG_FORMAT = "##Server##"; 22 | private static Logger LOG = LoggerFactory.getLogger(Server.class); 23 | 24 | public static void main(String args[]) { 25 | 26 | DateFormat df = new SimpleDateFormat("y-MM-dd HH:mm:ss"); 27 | Date date = new Date(); 28 | 29 | /* 30 | * Load parameter configs 31 | * */ 32 | Properties configs = Configs.CONFIGS; 33 | if (configs == null) { 34 | LOG.error(String.format("%s load config files failed", LOG_FORMAT).toString()); 35 | System.exit(1); 36 | } else { 37 | GlobalConfigs.GlobalConfigs = Configs.CONFIGS; 38 | } 39 | LOG.info(String.format("%s load configs files successfully.", LOG_FORMAT)); 40 | 41 | /* 42 | * Start Query Scheduler 43 | * */ 44 | ScheduledExecutorService service = Executors.newScheduledThreadPool(1); 45 | QueryScheduler scheduler = null; 46 | try { 47 | scheduler = new QueryScheduler(); 48 | long delay = Long.parseLong(configs.getProperty("POLL_INTERVAL")); 49 | service.scheduleWithFixedDelay(scheduler, 0, delay, TimeUnit.SECONDS); 50 | 51 | } catch (Exception e) { 52 | e.printStackTrace(); 53 | LOG.error(String.format("%s start query scheduler failed", LOG_FORMAT)); 54 | } 55 | 56 | LOG.info(String.format("%s start query scheduler successfully，current timestamp: %s", LOG_FORMAT, df.format(date))); 57 | 58 | /* 59 | * Start Listening Thread 60 | * */ 61 | SocketServer socketServer = null; 62 | int port = Integer.parseInt(configs.getProperty("PORT")); 63 | 64 | try { 65 | socketServer = new SocketServer(scheduler, port); 66 | socketServer.start(); 67 | } catch (Exception e) { 68 | e.printStackTrace(); 69 | LOG.error(String.format("%s SocketServer run failed. ", LOG_FORMAT)); 70 | } 71 | LOG.info(String.format("%s run socket server successfully, current timestamp: %s", LOG_FORMAT, df.format(date))); 72 | } 73 | } 74 | -------------------------------------------------------------------------------- /CDB-Server/crowdscheduler/src/main/java/com/tsinghua/dbgroup/crowddb/scheduler/server/SocketServer.java: -------------------------------------------------------------------------------- 1 | package com.tsinghua.dbgroup.crowddb.scheduler.server; 2 | 3 | import com.tsinghua.dbgroup.crowddb.scheduler.dispatcher.QueryScheduler; 4 | import com.tsinghua.dbgroup.crowddb.scheduler.utils.Util; 5 | 6 | import org.slf4j.Logger; 7 | import org.slf4j.LoggerFactory; 8 | 9 | import java.io.BufferedInputStream; 10 | import java.io.IOException; 11 | import java.net.ServerSocket; 12 | import java.net.Socket; 13 | 14 | /** 15 | * Created by talus on 7/3/16. 16 | */ 17 | public class SocketServer extends Thread{ 18 | 19 | private static String LOG_FORMAT = "##Server##"; 20 | private static Logger LOG = LoggerFactory.getLogger(SocketServer.class); 21 | 22 | private ServerSocket serverSocket; 23 | 24 | private QueryScheduler queryScheduler; 25 | 26 | public SocketServer(QueryScheduler queryScheduler, int port) throws IOException{ 27 | this.serverSocket = new ServerSocket(port); 28 | this.queryScheduler = queryScheduler; 29 | } 30 | 31 | @Override 32 | public void run() { 33 | while (true) { 34 | 35 | try { 36 | Socket socket = serverSocket.accept(); 37 | BufferedInputStream inputStream = new BufferedInputStream(socket.getInputStream()); 38 | 39 | String redData = Util.readFromSocket(inputStream); 40 | int queryId = -1; 41 | try { 42 | queryId = Integer.valueOf(redData); 43 | LOG.info(String.format("receive queryId = %d from socket stream", queryId)); 44 | 45 | this.queryScheduler.startNewQuery(queryId); 46 | } catch (NumberFormatException e) { 47 | LOG.error(String.format("get queryId failed, red data is @@@%s@@@", redData)); 48 | } catch (Exception e) { 49 | e.printStackTrace(); 50 | LOG.error(String.format("start new query failed, queryId = %d", queryId)); 51 | } 52 | } catch (Exception e) { 53 | LOG.error(LOG_FORMAT + "Cannot accept the connection or read from inputstream", e); 54 | } 55 | } 56 | } 57 | } 58 | -------------------------------------------------------------------------------- /CDB-Server/crowdscheduler/src/main/java/com/tsinghua/dbgroup/crowddb/scheduler/threads/ContinueQueryThread.java: -------------------------------------------------------------------------------- 1 | package com.tsinghua.dbgroup.crowddb.scheduler.threads; 2 | 3 | import com.tsinghua.dbgroup.crowddb.crowdexec.query.QueryManager; 4 | import com.tsinghua.dbgroup.crowddb.crowdexec.query.schema.Query; 5 | import org.slf4j.Logger; 6 | import org.slf4j.LoggerFactory; 7 | 8 | /** 9 | * Created by talus on 7/3/16. 10 | */ 11 | public class ContinueQueryThread extends Thread { 12 | 13 | private QueryManager queryManager; 14 | 15 | private Query query; 16 | 17 | private static Logger LOG = LoggerFactory.getLogger(ContinueQueryThread.class); 18 | 19 | public ContinueQueryThread(QueryManager queryManager, Query query) { 20 | setQueryManager(queryManager); 21 | setQuery(query); 22 | } 23 | 24 | public void setQueryManager(QueryManager queryManager) { 25 | this.queryManager = queryManager; 26 | } 27 | 28 | public void setQuery(Query query) { 29 | this.query = query; 30 | } 31 | 32 | @Override 33 | public void run() { 34 | try { 35 | queryManager.processNext(query); 36 | } catch (Exception e) { 37 | LOG.error("continue query execution failed.", e); 38 | } 39 | } 40 | } 41 | -------------------------------------------------------------------------------- /CDB-Server/crowdscheduler/src/main/java/com/tsinghua/dbgroup/crowddb/scheduler/threads/FinishQueryThread.java: -------------------------------------------------------------------------------- 1 | package com.tsinghua.dbgroup.crowddb.scheduler.threads; 2 | 3 | import com.tsinghua.dbgroup.crowddb.crowdexec.query.QueryManager; 4 | import com.tsinghua.dbgroup.crowddb.crowdexec.query.schema.Query; 5 | import org.slf4j.Logger; 6 | import org.slf4j.LoggerFactory; 7 | 8 | /** 9 | * Created by talus on 7/3/16. 10 | */ 11 | public class FinishQueryThread extends Thread { 12 | 13 | private static Logger LOG = LoggerFactory.getLogger(FinishQueryThread.class); 14 | 15 | private QueryManager queryManager; 16 | 17 | private Query query; 18 | 19 | public FinishQueryThread(QueryManager queryManager, Query query) { 20 | setQuery(query); 21 | setQueryManager(queryManager); 22 | } 23 | 24 | public void setQueryManager(QueryManager queryManager) { 25 | this.queryManager = queryManager; 26 | } 27 | 28 | public void setQuery(Query query) { 29 | this.query = query; 30 | } 31 | 32 | @Override 33 | public void run() { 34 | try { 35 | queryManager.finishQuery(query); 36 | } catch (Exception e) { 37 | LOG.error("finish query failed", e); 38 | } 39 | } 40 | } 41 | -------------------------------------------------------------------------------- /CDB-Server/crowdscheduler/src/main/java/com/tsinghua/dbgroup/crowddb/scheduler/threads/NewQueryThread.java: -------------------------------------------------------------------------------- 1 | package com.tsinghua.dbgroup.crowddb.scheduler.threads; 2 | 3 | import com.tsinghua.dbgroup.crowddb.crowdexec.query.QueryManager; 4 | import com.tsinghua.dbgroup.crowddb.crowdexec.query.schema.Query; 5 | import org.slf4j.Logger; 6 | import org.slf4j.LoggerFactory; 7 | 8 | /** 9 | * Created by talus on 7/3/16. 10 | */ 11 | public class NewQueryThread extends Thread { 12 | 13 | private QueryManager queryManager; 14 | 15 | private Query query; 16 | 17 | private static Logger LOG = LoggerFactory.getLogger(FinishQueryThread.class); 18 | 19 | public NewQueryThread(QueryManager queryManager, Query query) { 20 | setQueryManager(queryManager); 21 | setQuery(query); 22 | } 23 | 24 | public void setQueryManager(QueryManager queryManager) { 25 | this.queryManager = queryManager; 26 | } 27 | 28 | public void setQuery(Query query) { 29 | this.query = query; 30 | } 31 | 32 | @Override 33 | public void run() { 34 | //TODO: start new Query 35 | try { 36 | queryManager.startNewQuery(query); 37 | } catch (Exception e) { 38 | LOG.error("query scheduler start new query failed.", e); 39 | } 40 | } 41 | } 42 | -------------------------------------------------------------------------------- /CDB-Server/crowdscheduler/src/main/java/com/tsinghua/dbgroup/crowddb/scheduler/utils/Util.java: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright (C) 2016-2016 by The Department of Computer Science and 3 | * Technology, Tsinghua University 4 | * 5 | * Redistribution of this file is permitted under the terms of 6 | * the BSD license. 7 | * 8 | * Author : XuepingWeng 9 | * Created : 10/12/16 11:38 PM 10 | * Modified : 11 | * Contact : wxping715@gmail.com 12 | */ 13 | 14 | package com.tsinghua.dbgroup.crowddb.scheduler.utils; 15 | 16 | import java.io.BufferedInputStream; 17 | import java.io.IOException; 18 | 19 | public class Util { 20 | 21 | public static String readFromSocket(BufferedInputStream inputStream) { 22 | int red = -1; 23 | byte[] buffer = new byte[8 * 1024]; 24 | byte[] redData; 25 | 26 | StringBuilder clientData = new StringBuilder(); 27 | String redDataText; 28 | 29 | if (inputStream == null) return null; 30 | 31 | try { 32 | while ((red = inputStream.read(buffer)) > -1) { 33 | redData = new byte[red]; 34 | System.arraycopy(buffer, 0, redData, 0, red); 35 | redDataText = new String(redData, "UTF-8"); 36 | clientData.append(redDataText); 37 | } 38 | } catch (IOException e) { 39 | e.printStackTrace(); 40 | return null; 41 | } finally { 42 | return clientData.toString(); 43 | } 44 | } 45 | } 46 | -------------------------------------------------------------------------------- /CDB-Server/crowdscheduler/src/main/resources/logback.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | %d{HH:mm:ss.SSS} %highlight(%-5level) %cyan(%logger{15}) - %msg%n 7 | 8 | 9 | 10 | 11 | 12 | 13 | ./logs/crowddb-%d{yyyy-MM-dd}.log 14 | 15 | 16 | 1 17 | 18 | 19 | %d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | -------------------------------------------------------------------------------- /CDB-Server/crowdscheduler/src/test/java/com/tsinghua/dbgroup/crowddb/scheduler/dispatcher/QuerySchedulerTest.java: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright (C) 2016-2016 by The Department of Computer Science and 3 | * Technology, Tsinghua University 4 | * 5 | * Redistribution of this file is permitted under the terms of 6 | * the BSD license. 7 | * 8 | * Author : XuepingWeng 9 | * Created : 10/13/16 12:36 AM 10 | * Modified : 11 | * Contact : wxping715@gmail.com 12 | */ 13 | 14 | package com.tsinghua.dbgroup.crowddb.scheduler.dispatcher; 15 | 16 | import org.junit.Test; 17 | 18 | import static org.junit.Assert.*; 19 | 20 | public class QuerySchedulerTest { 21 | @Test 22 | public void TestNewQuery() throws Exception { 23 | int queryId = 1; 24 | QueryScheduler scheduler = new QueryScheduler(); 25 | scheduler.startNewQuery(queryId); 26 | } 27 | 28 | } -------------------------------------------------------------------------------- /CDB-Server/crowdscheduler/src/test/java/com/tsinghua/dbgroup/crowddb/scheduler/server/SocketServerTest.java: -------------------------------------------------------------------------------- 1 | package com.tsinghua.dbgroup.crowddb.scheduler.server; 2 | import com.tsinghua.dbgroup.crowddb.scheduler.server.SocketServer; 3 | import junit.framework.TestCase; 4 | 5 | 6 | import java.io.BufferedOutputStream; 7 | import java.io.DataOutputStream; 8 | import java.net.Socket; 9 | 10 | /** 11 | * Created by talus on 7/3/16. 12 | */ 13 | public class SocketServerTest extends TestCase { 14 | 15 | public void testSocketServer() { 16 | 17 | try { 18 | SocketServer server = new SocketServer(null, 1234); 19 | server.start(); 20 | 21 | Socket socket= new Socket("127.0.0.1",1234); 22 | BufferedOutputStream outputStream = new BufferedOutputStream(socket.getOutputStream()); 23 | DataOutputStream dataOutputStream = new DataOutputStream(outputStream); 24 | 25 | dataOutputStream.writeInt(12); 26 | dataOutputStream.flush(); 27 | dataOutputStream.close(); 28 | socket.close(); 29 | 30 | } catch (Exception e) { 31 | e.printStackTrace(); 32 | } 33 | } 34 | } 35 | -------------------------------------------------------------------------------- /CDB-Server/crowdsql/pom.xml: -------------------------------------------------------------------------------- 1 | 3 | 4 | crowddb 5 | com.tsinghua.dbgroup.crowddb 6 | 0.01 7 | 8 | 4.0.0 9 | 10 | crowdsql 11 | jar 12 | 13 | crowdsql 14 | http://maven.apache.org 15 | 16 | 17 | UTF-8 18 | 19 | 20 | 21 | 22 | 23 | org.apache.maven.plugins 24 | maven-compiler-plugin 25 | 2.3.1 26 | 27 | 1.8 28 | 1.8 29 | UTF-8 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | junit 38 | junit 39 | 4.12 40 | test 41 | 42 | 43 | org.apache.commons 44 | commons-lang3 45 | 3.5 46 | 47 | 48 | com.tsinghua.dbgroup.crowddb 49 | crowdstorage 50 | 0.01 51 | 52 | 53 | 54 | -------------------------------------------------------------------------------- /CDB-Server/crowdsql/src/main/java/META-INF/MANIFEST.MF: -------------------------------------------------------------------------------- 1 | Manifest-Version: 1.0 2 | Main-Class: com.tsinghua.dbgroup.crowddb.scheduler.Server 3 | 4 | -------------------------------------------------------------------------------- /CDB-Server/crowdsql/src/main/java/com/tsinghua/dbgroup/crowddb/crowdsql/operator/OperatorHelper.java: -------------------------------------------------------------------------------- 1 | package com.tsinghua.dbgroup.crowddb.crowdsql.operator; 2 | 3 | import org.slf4j.Logger; 4 | import org.slf4j.LoggerFactory; 5 | 6 | import java.lang.reflect.Field; 7 | import java.util.HashSet; 8 | import java.util.Set; 9 | 10 | public class OperatorHelper { 11 | 12 | static private Set crowdOperatorSet = new HashSet(); 13 | 14 | static private Set operatorSet = new HashSet<>(); 15 | 16 | static private Logger logger = LoggerFactory.getLogger(OperatorHelper.class); 17 | 18 | static public Set getCrowdOperatorSet() { 19 | return crowdOperatorSet; 20 | } 21 | 22 | static public Set getOperatorSet() { 23 | return operatorSet; 24 | } 25 | 26 | static { 27 | generateCrowdOperatorSet(); 28 | generateKeywordSet(); 29 | } 30 | 31 | private static void generateCrowdOperatorSet() { 32 | Field[] fields = Operators.class.getDeclaredFields(); 33 | Operators operators = new Operators(); 34 | for (Field field : fields) { 35 | field.setAccessible(true); 36 | try { 37 | if (field.getName().startsWith("CROWD")) { 38 | String fieldValue = field.get(operators).toString(); 39 | crowdOperatorSet.add(fieldValue); 40 | } 41 | } catch (IllegalAccessException e) { 42 | e.printStackTrace(); 43 | } 44 | } 45 | crowdOperatorSet.add(Operators.FILL); 46 | crowdOperatorSet.add(Operators.COLLECT); 47 | } 48 | 49 | private static void generateKeywordSet() { 50 | Field[] fields = Operators.class.getDeclaredFields(); 51 | Operators operators = new Operators(); 52 | for (Field field : fields) { 53 | field.setAccessible(true); 54 | try { 55 | String fieldValue = field.get(operators).toString(); 56 | operatorSet.add(fieldValue); 57 | } catch (IllegalAccessException e) { 58 | e.printStackTrace(); 59 | } 60 | } 61 | } 62 | 63 | } 64 | -------------------------------------------------------------------------------- /CDB-Server/crowdsql/src/main/java/com/tsinghua/dbgroup/crowddb/crowdsql/operator/Operators.java: -------------------------------------------------------------------------------- 1 | package com.tsinghua.dbgroup.crowddb.crowdsql.operator; 2 | 3 | public class Operators { 4 | // Variable name of crowd operator must start with CROWD, real value can be any string 5 | public final static String CROWD_EQ = "CROWD_EQ"; 6 | public final static String CROWD_GT = "CROWD_GT"; 7 | public final static String CROWD_GTE = "CROWD_GTE"; 8 | public final static String CROWD_LT = "CROWD_LT"; 9 | public final static String CROWD_IN = "CROWD_IN"; 10 | public final static String CROWD_BETWEEN = "CROWD_BETWEEN"; 11 | public final static String CROWD_LIKE = "CROWD_LIKE"; 12 | 13 | 14 | public final static String SELECT = "SELECT"; 15 | public final static String SINGLELABEL = "SINGLELABEL"; 16 | public final static String MULTILABEL = "MULTILABEL"; 17 | public final static String COLLECT = "COLLECT"; 18 | public final static String FILL = "FILL"; 19 | public final static String FROM = "FROM"; 20 | public final static String WHERE = "WHERE"; 21 | public final static String ON = "ON"; 22 | public final static String TAGIN = "TAGIN"; 23 | public final static String IN = "IN"; 24 | public final static String BETWEEN = "BETWEEN"; 25 | public final static String LIMIT = "LIMIT"; 26 | 27 | public final static String AND = "AND"; 28 | public final static String OR = "OR"; 29 | 30 | } 31 | -------------------------------------------------------------------------------- /CDB-Server/crowdsql/src/main/java/com/tsinghua/dbgroup/crowddb/crowdsql/parser/ISqlParser.java: -------------------------------------------------------------------------------- 1 | package com.tsinghua.dbgroup.crowddb.crowdsql.parser; 2 | 3 | import com.tsinghua.dbgroup.crowddb.crowdsql.query.SqlContext; 4 | 5 | public interface ISqlParser { 6 | SqlContext parse(String query,String dbName); 7 | } 8 | -------------------------------------------------------------------------------- /CDB-Server/crowdsql/src/main/java/com/tsinghua/dbgroup/crowddb/crowdsql/tree/NodeType.java: -------------------------------------------------------------------------------- 1 | package com.tsinghua.dbgroup.crowddb.crowdsql.tree; 2 | 3 | public enum NodeType { 4 | // CROWD_GT, 5 | // CROWD_GTE, 6 | // CROWD_LT, 7 | EXECUTE_SQL, 8 | JOIN, 9 | CROWD_EQ, 10 | CROWD_JOIN, 11 | CROWD_GT, 12 | CROWD_LT, 13 | CROWD_IN, 14 | COLLECT, 15 | FILL, 16 | SINGLELABEL, 17 | MULTILABEL, 18 | EQ, 19 | GT, 20 | LT, 21 | IN, 22 | AND, 23 | OR, 24 | PROJECT, 25 | GRAPH_MODEL, 26 | } 27 | -------------------------------------------------------------------------------- /CDB-Server/crowdsql/src/main/java/com/tsinghua/dbgroup/crowddb/crowdsql/tree/SqlTree.java: -------------------------------------------------------------------------------- 1 | package com.tsinghua.dbgroup.crowddb.crowdsql.tree; 2 | 3 | import java.util.LinkedList; 4 | import java.util.Queue; 5 | 6 | public class SqlTree { 7 | 8 | private SqlTreeNode root; 9 | 10 | public SqlTree(SqlTreeNode root) { 11 | this.root = root; 12 | } 13 | 14 | public SqlTreeNode getRoot() { 15 | return root; 16 | } 17 | 18 | public void setRoot(SqlTreeNode root) { 19 | this.root = root; 20 | } 21 | 22 | @Override 23 | public String toString() { 24 | Queue nodeQueue = new LinkedList<>(); 25 | nodeQueue.add(root); 26 | StringBuilder objectStr = new StringBuilder(); 27 | objectStr.append(System.lineSeparator()); 28 | while (!nodeQueue.isEmpty()) { 29 | SqlTreeNode node = nodeQueue.poll(); 30 | objectStr.append(render(node)).append(System.lineSeparator()); 31 | if (node.left != null) { 32 | nodeQueue.add(node.left); 33 | } 34 | if (node.right != null) { 35 | nodeQueue.add(node.right); 36 | } 37 | } 38 | return objectStr.toString(); 39 | } 40 | 41 | String render(SqlTreeNode node) { 42 | return node + " SQL: id " +node.getNodeID()+" "+ node.query + " nodeType: " + node.nodeType + " left: " + node.left + " right: " + node.right; 43 | } 44 | } 45 | -------------------------------------------------------------------------------- /CDB-Server/crowdsql/src/main/java/com/tsinghua/dbgroup/crowddb/crowdsql/tree/SqlTreeNode.java: -------------------------------------------------------------------------------- 1 | package com.tsinghua.dbgroup.crowddb.crowdsql.tree; 2 | 3 | import com.tsinghua.dbgroup.crowddb.crowdsql.query.SqlContext; 4 | 5 | import java.util.ArrayList; 6 | import java.util.List; 7 | 8 | public class SqlTreeNode { 9 | String whereClause; 10 | String query; 11 | NodeType nodeType; 12 | SqlTreeNode left; 13 | SqlTreeNode right; 14 | SqlContext sqlContext; 15 | private String tableName = null; 16 | int nodeID; 17 | List projects = new ArrayList<>(); 18 | List froms = new ArrayList<>(); 19 | 20 | public SqlTreeNode() { 21 | 22 | } 23 | 24 | public SqlTreeNode(NodeType nodeType, SqlContext sqlContext,int id) { 25 | this.nodeType = nodeType; 26 | this.sqlContext = sqlContext; 27 | nodeID = id; 28 | } 29 | 30 | public SqlTreeNode(NodeType nodeType, SqlContext sqlContext, String query) { 31 | this.nodeType = nodeType; 32 | this.sqlContext = sqlContext; 33 | this.query = query; 34 | } 35 | 36 | public String getWhereClause() { 37 | return whereClause; 38 | } 39 | 40 | public void setWhereClause(String whereClause) { 41 | this.whereClause = whereClause; 42 | } 43 | 44 | public String getQuery() { 45 | return query; 46 | } 47 | 48 | public void setQuery(String query) { 49 | this.query = query; 50 | } 51 | 52 | public NodeType getNodeType() { 53 | return nodeType; 54 | } 55 | 56 | public void setNodeType(NodeType nodeType) { 57 | this.nodeType = nodeType; 58 | } 59 | 60 | public SqlTreeNode getLeft() { 61 | return left; 62 | } 63 | 64 | public void setLeft(SqlTreeNode left) { 65 | this.left = left; 66 | } 67 | 68 | public SqlTreeNode getRight() { 69 | return right; 70 | } 71 | 72 | public void setRight(SqlTreeNode right) { 73 | this.right = right; 74 | } 75 | 76 | public SqlContext getSqlContext() { 77 | return sqlContext; 78 | } 79 | 80 | public void setSqlContext(SqlContext sqlContext) { 81 | this.sqlContext = sqlContext; 82 | } 83 | 84 | public List getProjects() { 85 | return projects; 86 | } 87 | 88 | public void setProjects(List projects) { 89 | this.projects = projects; 90 | } 91 | 92 | public List getFroms() { 93 | return froms; 94 | } 95 | 96 | public void setFroms(List froms) { 97 | this.froms = froms; 98 | } 99 | 100 | public String getTableName() { 101 | return tableName; 102 | } 103 | 104 | public void setTableName(String tableName) { 105 | this.tableName = tableName; 106 | } 107 | 108 | public int getNodeID() { 109 | return nodeID; 110 | } 111 | 112 | public void setNodeID(int nodeID) { 113 | this.nodeID = nodeID; 114 | } 115 | 116 | @Override 117 | public String toString() { 118 | return "@" + Integer.toHexString(hashCode()); 119 | } 120 | } 121 | 122 | 123 | 124 | -------------------------------------------------------------------------------- /CDB-Server/crowdsql/src/main/java/com/tsinghua/dbgroup/crowddb/crowdsql/util/Utils.java: -------------------------------------------------------------------------------- 1 | package com.tsinghua.dbgroup.crowddb.crowdsql.util; 2 | 3 | public class Utils { 4 | public static boolean isAllWhiteSpace(String s) { 5 | for (int i = 0; i < s.length(); ++i) { 6 | if (!Character.isWhitespace(s.charAt(i))) { 7 | return false; 8 | } 9 | } 10 | return true; 11 | } 12 | 13 | public static boolean matchLogicOperator(String s, int i, String subStr) { 14 | return i + subStr.length() <= s.length() && s.substring(i, i+subStr.length()).equals(subStr) && 15 | (i == 0 || isSeprator(s.charAt(i-1))) && 16 | (i+1 < s.length() || isSeprator(s.charAt(i+1))); 17 | } 18 | 19 | public static boolean isSeprator(char ch) { 20 | return ch == '(' || ch == ')' || Character.isWhitespace(ch); 21 | } 22 | 23 | public static String packString(String[] strings,int start,int end){ 24 | String pack = null; 25 | for (int i = start; i < end; ++i){ 26 | pack += strings[i]; 27 | } 28 | return pack; 29 | } 30 | } 31 | -------------------------------------------------------------------------------- /CDB-Server/crowdsql/src/test/java/com/tsinghua/dbgroup/crowddb/crowdsql/query/SqlContextTest.java: -------------------------------------------------------------------------------- 1 | package com.tsinghua.dbgroup.crowddb.crowdsql.query; 2 | 3 | import org.junit.Test; 4 | 5 | import static org.junit.Assert.*; 6 | 7 | /** 8 | * Created by x-yu13 on 16/10/10. 9 | */ 10 | public class SqlContextTest { 11 | @Test 12 | public void getWheres() throws Exception { 13 | 14 | } 15 | 16 | @Test 17 | public void setWheres() throws Exception { 18 | 19 | } 20 | 21 | } -------------------------------------------------------------------------------- /CDB-Server/crowdstorage/pom.xml: -------------------------------------------------------------------------------- 1 | 3 | 4 | crowddb 5 | com.tsinghua.dbgroup.crowddb 6 | 0.01 7 | 8 | 4.0.0 9 | 10 | crowdstorage 11 | jar 12 | 13 | crowdstorage 14 | http://maven.apache.org 15 | 16 | 17 | UTF-8 18 | 19 | 20 | 21 | 22 | com.tsinghua.dbgroup.crowddb 23 | crowdcore 24 | 0.01 25 | 26 | 27 | junit 28 | junit 29 | 4.12 30 | test 31 | 32 | 33 | commons-dbutils 34 | commons-dbutils 35 | 1.6 36 | 37 | 38 | org.mariadb.jdbc 39 | mariadb-java-client 40 | 1.5.4 41 | 42 | 43 | mysql 44 | mysql-connector-java 45 | 6.0.4 46 | jar 47 | compile 48 | 49 | 50 | 51 | -------------------------------------------------------------------------------- /CDB-Server/crowdstorage/src/main/java/com/tsinghua/dbgroup/crowddb/crowdstorage/table/BaseDBStorage.java: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright (C) 2016-2016 by The Department of Computer Science and 3 | * Technology, Tsinghua University 4 | * 5 | * Redistribution of this file is permitted under the terms of 6 | * the BSD license. 7 | * 8 | * Author : XuepingWeng 9 | * Created : 11/3/16 12:51 PM 10 | * Modified : 11 | * Contact : wxping715@gmail.com 12 | */ 13 | 14 | package com.tsinghua.dbgroup.crowddb.crowdstorage.table; 15 | import com.tsinghua.dbgroup.crowddb.crowdcore.configs.GlobalConfigs; 16 | 17 | import org.slf4j.Logger; 18 | import org.slf4j.LoggerFactory; 19 | 20 | import java.io.InputStream; 21 | import java.sql.Connection; 22 | import java.sql.DriverManager; 23 | import java.util.Properties; 24 | 25 | /** 26 | * Created by talus on 16/6/8. 27 | */ 28 | public class BaseDBStorage { 29 | 30 | private static String LOG_FORMAT = "##BaseDBStorage##"; 31 | 32 | private static Logger LOG = LoggerFactory.getLogger(BaseDBStorage.class); 33 | 34 | protected Connection getConnnection() { 35 | String host, port, user, password, connUrl; 36 | try { 37 | host = (String) GlobalConfigs.GlobalConfigs.getOrDefault("HOST", "127.0.0.1"); 38 | port = (String) GlobalConfigs.GlobalConfigs.getOrDefault("MYSQL_PORT", "3306"); 39 | user = (String) GlobalConfigs.GlobalConfigs.getOrDefault("USERNAME", "3306"); 40 | password = (String) GlobalConfigs.GlobalConfigs.getOrDefault("PASSWORD", "3306"); 41 | // host = "166.111.71.172"; 42 | // port = "3306"; 43 | // user = "crowddb"; 44 | // password = "crowddb!password"; 45 | } catch (Exception e) { 46 | LOG.error(LOG_FORMAT, " load config file failed. ", e); 47 | return null; 48 | } 49 | 50 | connUrl = String.format("jdbc:mysql://%s:%s/crowddb_temp?characterEncoding=UTF-8", 51 | host, port); 52 | 53 | Connection conn = null; 54 | try { 55 | Class.forName("com.mysql.jdbc.Driver"); 56 | conn = DriverManager.getConnection(connUrl, user, password); 57 | } catch (Exception e) { 58 | LOG.error(LOG_FORMAT+" connect the database failed.", e); 59 | } 60 | return conn; 61 | } 62 | 63 | protected Properties loadDBConfig() throws Exception { 64 | Properties prop = new Properties(); 65 | InputStream input = null; 66 | String configName = "jdbc.properties.development"; 67 | String connectionUrl = ""; 68 | 69 | input = BaseDBStorage.class.getClassLoader().getResourceAsStream(configName); 70 | if (input == null) { 71 | throw new Exception("cann't find the config file"); 72 | } 73 | prop.load(input); 74 | return prop; 75 | } 76 | } 77 | -------------------------------------------------------------------------------- /CDB-Server/crowdstorage/src/main/java/com/tsinghua/dbgroup/crowddb/crowdstorage/table/ITableManager.java: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright (C) 2016-2016 by The Department of Computer Science and 3 | * Technology, Tsinghua University 4 | * 5 | * Redistribution of this file is permitted under the terms of 6 | * the BSD license. 7 | * 8 | * Author : XuepingWeng 9 | * Created : 11/3/16 12:51 PM 10 | * Modified : 11 | * Contact : wxping715@gmail.com 12 | */ 13 | 14 | package com.tsinghua.dbgroup.crowddb.crowdstorage.table; 15 | 16 | /** 17 | * Created by talus on 16/6/2. 18 | */ 19 | public interface ITableManager { 20 | 21 | } 22 | -------------------------------------------------------------------------------- /CDB-Server/crowdstorage/src/main/java/com/tsinghua/dbgroup/crowddb/crowdstorage/table/schema/ColumnsSchema.java: -------------------------------------------------------------------------------- 1 | package com.tsinghua.dbgroup.crowddb.crowdstorage.table.schema; 2 | 3 | import java.util.HashMap; 4 | 5 | /** 6 | * Created by talus on 11/15/16. 7 | */ 8 | public class ColumnsSchema { 9 | 10 | public static final int SCHEMA_ID = 0x02; 11 | 12 | private String id; 13 | 14 | private HashMap columns; 15 | 16 | public ColumnsSchema(String id, HashMap columns) { 17 | this.id = id; 18 | this.columns = columns; 19 | } 20 | 21 | public String getId() { 22 | return id; 23 | } 24 | 25 | public void setId(String id) { 26 | this.id = id; 27 | } 28 | 29 | public HashMap getColumns() { 30 | return columns; 31 | } 32 | 33 | public void setColumns(HashMap columns) { 34 | this.columns = columns; 35 | } 36 | } 37 | -------------------------------------------------------------------------------- /CDB-Server/crowdstorage/src/main/java/com/tsinghua/dbgroup/crowddb/crowdstorage/table/schema/EqualSchema.java: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright (C) 2016-2016 by The Department of Computer Science and 3 | * Technology, Tsinghua University 4 | * 5 | * Redistribution of this file is permitted under the terms of 6 | * the BSD license. 7 | * 8 | * Author : XuepingWeng 9 | * Created : 11/3/16 12:51 PM 10 | * Modified : 11 | * Contact : wxping715@gmail.com 12 | */ 13 | 14 | package com.tsinghua.dbgroup.crowddb.crowdstorage.table.schema; 15 | /** 16 | * Created by talus on 16/6/8. 17 | */ 18 | public class EqualSchema { 19 | 20 | public static String createSQL = new StringBuilder() 21 | .append("CREATE TABLE %s (") 22 | .append("eid1 int NOT NULL,") 23 | .append("res INT NOT NULL") 24 | .append(")").toString(); 25 | 26 | public static String insertSQL = new StringBuilder() 27 | .append("INSERT INTO %s ") 28 | .append("(`eid1`, `res`) values ") 29 | .append("%s").toString(); 30 | 31 | 32 | public static final int SCHEMA_ID = 0x02; 33 | 34 | private int id; 35 | 36 | private String eid; 37 | 38 | private int res; 39 | 40 | public EqualSchema() { 41 | 42 | } 43 | 44 | public EqualSchema(String eid, int res) { 45 | this.eid = eid; 46 | this.res = res; 47 | } 48 | 49 | public int getId() { 50 | return id; 51 | } 52 | 53 | public void setId(int id) { 54 | this.id = id; 55 | } 56 | 57 | public String getEid() { 58 | return eid; 59 | } 60 | 61 | public void setEid(String eid) { 62 | this.eid = eid; 63 | } 64 | 65 | public int getRes() { 66 | return res; 67 | } 68 | 69 | public void setRes(int res) { 70 | this.res = res; 71 | } 72 | } 73 | -------------------------------------------------------------------------------- /CDB-Server/crowdstorage/src/main/java/com/tsinghua/dbgroup/crowddb/crowdstorage/table/schema/JoinSchema.java: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright (C) 2016-2016 by The Department of Computer Science and 3 | * Technology, Tsinghua University 4 | * 5 | * Redistribution of this file is permitted under the terms of 6 | * the BSD license. 7 | * 8 | * Author : XuepingWeng 9 | * Created : 11/3/16 12:51 PM 10 | * Modified : 11 | * Contact : wxping715@gmail.com 12 | */ 13 | 14 | package com.tsinghua.dbgroup.crowddb.crowdstorage.table.schema; 15 | 16 | /** 17 | * Created by talus on 16/6/8. 18 | */ 19 | public class JoinSchema { 20 | 21 | public static String createSQL = new StringBuilder() 22 | .append("CREATE TABLE %s (") 23 | .append("eid1 int NOT NULL,") 24 | .append("eid2 int NOT NULL,") 25 | .append("res INT NOT NULL") 26 | .append(")").toString(); 27 | 28 | public static String insertSQL = new StringBuilder() 29 | .append("INSERT INTO %s ") 30 | .append("(`eid1`, `eid2`, `res`) values %s").toString(); 31 | 32 | public static final int SCHEMA_ID = 0x01; 33 | 34 | private int id; 35 | 36 | private String eid1; 37 | 38 | private String eid2; 39 | 40 | private int res; 41 | 42 | public JoinSchema() { 43 | 44 | } 45 | 46 | public JoinSchema(String eid1, String eid2, int res) { 47 | this.eid1 = eid1; 48 | this.eid2 = eid2; 49 | this.res = res; 50 | } 51 | 52 | public int getId() { 53 | return id; 54 | } 55 | 56 | public void setId(int id) { 57 | this.id = id; 58 | } 59 | 60 | public String getEid1() { 61 | return eid1; 62 | } 63 | 64 | public void setEid1(String eid1) { 65 | this.eid1 = eid1; 66 | } 67 | 68 | public String getEid2() { 69 | return eid2; 70 | } 71 | 72 | public void setEid2(String eid2) { 73 | this.eid2 = eid2; 74 | } 75 | 76 | public int getRes() { 77 | return res; 78 | } 79 | 80 | public void setRes(int res) { 81 | this.res = res; 82 | } 83 | } 84 | -------------------------------------------------------------------------------- /CDB-Server/crowdstorage/src/main/java/com/tsinghua/dbgroup/crowddb/crowdstorage/utils/TableHelper.java: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright (C) 2016-2016 by The Department of Computer Science and 3 | * Technology, Tsinghua University 4 | * 5 | * Redistribution of this file is permitted under the terms of 6 | * the BSD license. 7 | * 8 | * Author : XuepingWeng 9 | * Created : 11/3/16 12:52 PM 10 | * Modified : 11 | * Contact : wxping715@gmail.com 12 | */ 13 | 14 | package com.tsinghua.dbgroup.crowddb.crowdstorage.utils; 15 | 16 | import com.tsinghua.dbgroup.crowddb.crowdstorage.table.TableManager; 17 | import com.tsinghua.dbgroup.crowddb.crowdstorage.table.schema.EqualSchema; 18 | import com.tsinghua.dbgroup.crowddb.crowdstorage.table.schema.JoinSchema; 19 | import org.apache.commons.lang3.tuple.Pair; 20 | 21 | import java.util.UUID; 22 | 23 | /** 24 | * Created by talus on 16/6/11. 25 | */ 26 | public class TableHelper { 27 | 28 | public static String generateTmpTableName(int queryId, int taskId) { 29 | String randomStr = UUID.randomUUID().toString().substring(0,8); 30 | return String.format("q%d_t%d_%s", queryId, taskId, randomStr); 31 | } 32 | 33 | public static Class getSchemaClass(int tableType) { 34 | Object object = null; 35 | 36 | switch (tableType) { 37 | case JoinSchema.SCHEMA_ID: 38 | object = (JoinSchema)new JoinSchema(); 39 | break; 40 | 41 | case EqualSchema.SCHEMA_ID: 42 | object = (EqualSchema)new EqualSchema(); 43 | break; 44 | } 45 | return object.getClass(); 46 | } 47 | 48 | public static String wrapTableName(String tableName) { 49 | return String.format("%s.%s", TableManager.TMP_DATABASE, tableName); 50 | } 51 | 52 | public static Pair unpackTableColumn(String column) { 53 | int index = column.lastIndexOf('.'); 54 | if (index == -1) return null; 55 | 56 | String col = column.substring(index+1,column.length()-1); 57 | String tableName = column.substring(0, index-1); 58 | // Pair pair = new Pair(tableName, col); 59 | // return pair; 60 | return null; 61 | } 62 | } 63 | -------------------------------------------------------------------------------- /CDB-Server/crowdstorage/src/main/java/com/tsinghua/dbgroup/crowddb/crowdstorage/utils/Utils.java: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright (C) 2016-2016 by The Department of Computer Science and 3 | * Technology, Tsinghua University 4 | * 5 | * Redistribution of this file is permitted under the terms of 6 | * the BSD license. 7 | * 8 | * Author : XuepingWeng 9 | * Created : 11/3/16 12:53 PM 10 | * Modified : 11 | * Contact : wxping715@gmail.com 12 | */ 13 | 14 | package com.tsinghua.dbgroup.crowddb.crowdstorage.utils; 15 | 16 | public class Utils { 17 | 18 | static String Separator = "\\."; 19 | 20 | public static String packetColumn(String table, String column) { 21 | String[] items = column.split(Utils.Separator); 22 | if (items.length > 1) { 23 | return column; 24 | } 25 | return table + "." + column; 26 | } 27 | 28 | public static String unpacketColumn(String column) { 29 | String[] items = column.split(Utils.Separator); 30 | if (items.length > 1) { 31 | return items[1]; 32 | } 33 | return column; 34 | } 35 | 36 | 37 | public static String packetTable(String dbName, String tableName) { 38 | String[] items = tableName.split(Utils.Separator); 39 | if (items.length > 1) { 40 | return tableName; 41 | } 42 | return dbName + "." + tableName; 43 | } 44 | 45 | public static String unpacketTable(String table) { 46 | String[] items = table.split(Utils.Separator); 47 | if (items.length > 1) { 48 | return items[1]; 49 | } 50 | return table; 51 | } 52 | 53 | public static String unpacketDBName(String table) { 54 | String[] items = table.split(Utils.Separator); 55 | if (items.length > 1) { 56 | return items[0]; 57 | } 58 | return null; 59 | } 60 | } 61 | 62 | 63 | 64 | -------------------------------------------------------------------------------- /CDB-Server/crowdstorage/src/main/resources/jdbc.properties.development: -------------------------------------------------------------------------------- 1 | ## MySQL 2 | # MYSQL_SOURCE=jdbc:mysql://166.111.71.172:3306/crowddb_temp?useUnicode=true&characterEncoding=UTF-8 3 | # USERNAME=crowddb 4 | # PASSWORD=crowddb!password -------------------------------------------------------------------------------- /CDB-Server/resources/META-INF/MANIFEST.MF: -------------------------------------------------------------------------------- 1 | Manifest-Version: 1.0 2 | Main-Class: com.tsinghua.dbgroup.crowddb.scheduler.Server 3 | 4 | -------------------------------------------------------------------------------- /CDB-Server/resources/logback.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | %d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n 7 | 8 | 9 | 10 | 11 | 12 | 13 | ./logs/crowddb-%d{yyyy-MM-dd}.log 14 | 15 | 16 | 1 17 | 18 | 19 | %d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | -------------------------------------------------------------------------------- /CDB-Server/resources/runtime.configs.properties: -------------------------------------------------------------------------------- 1 | ### Database settings 2 | 3 | ## MySQL 4 | 5 | USERNAME = [username] 6 | PASSWORD = [password] 7 | HOST = 127.0.0.1 8 | MYSQL_PORT = 3306 9 | 10 | ### Server Settings 11 | 12 | # Server listening port for receiving query request 13 | PORT = 1234 14 | 15 | # The time interval of rolling queries 16 | POLL_INTERVAL = 300 17 | 18 | ### Crowdsourcing Platform Settings 19 | 20 | # the default crowdsourcing platform for answering questions, there are three options: 21 | # 1. CC = ChinaCrowds, http://chinacrowds.com/ 22 | # 2. CF = Crowdflower, http://www.crowdflower.com/ 23 | # 3. AMT = Amazon Mechanical Turk, https://www.mturk.com/mturk/welcome 24 | DEFAULT_PLATFORM = CC 25 | 26 | # rewards for each hit or page, it is different by different platforms, please view the relevant platform 27 | REWARD = 0.01 28 | 29 | # max assignment number for each hit 30 | MAX_ASSIGNMENT = 5 31 | 32 | # assignment server 33 | ASSIGNMENT_SERVER_URL = http://127.0.0.1:9000 34 | 35 | # simulation settings 36 | SIMULATION = false 37 | -------------------------------------------------------------------------------- /CDB-Server/scripts/runing_scripts/init_db.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | #HOST="166.111.71.172" 4 | HOST="127.0.0.1" 5 | PORT="3306" 6 | USER="root" 7 | PASSWORD="" 8 | 9 | mysql -h$HOST -P$PORT -u$USER -p$PASSWORD -e `dirname $0`/init_db.sql 10 | -------------------------------------------------------------------------------- /CDB-Server/scripts/runing_scripts/init_db.sql: -------------------------------------------------------------------------------- 1 | CREATE USER crowddb IDENTIFIED by 'crowddb!password'; 2 | 3 | CREATE database crowddb_meta DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; 4 | CREATE database crowddb_temp DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; 5 | CREATE database crowddb_assign DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; 6 | CREATE database crowddb_assign DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; 7 | 8 | USE crowddb_meta; 9 | CREATE TABLE IF NOT EXISTS `query` ( 10 | `id` int(11) NOT NULL AUTO_INCREMENT, 11 | `qsql` varchar(1024) DEFAULT NULL, 12 | `timestamp` datetime NOT NULL, 13 | `status` varchar(20) NOT NULL, 14 | `user_id` int(11) NOT NULL, 15 | `current_sqlnode_id` int(11) NOT NULL, 16 | `result_table` varchar(20) NOT NULL, 17 | `db_name` varchar(255) DEFAULT NULL, 18 | `error_message` varchar(255) DEFAULT NULL, 19 | `task_title` varchar(255) DEFAULT NULL, 20 | `platform` varchar(20) DEFAULT NULL, 21 | `gmodel` varchar(1) DEFAULT NULL, 22 | `process` FLOAT DEFAULT 0, 23 | PRIMARY KEY (`id`) 24 | ) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=16 ; 25 | 26 | GRANT ALL PRIVILEGES ON *.* TO crowddb; 27 | FLUSH PRIVILEGES; 28 | -------------------------------------------------------------------------------- /CDB-Server/scripts/test_scripts/send_request.py: -------------------------------------------------------------------------------- 1 | import sys 2 | import socket 3 | 4 | host = "127.0.0.1" 5 | port = 1234 6 | 7 | def send_request(): 8 | s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 9 | print "start socket server" 10 | s.connect((host, port)) 11 | print "bind {}:{}".format(host, port) 12 | 13 | if len(sys.argv) != 2: 14 | print "need one param, python send_request.py [idx]" 15 | queryId = sys.argv[1] 16 | s.send(queryId) 17 | print "send {}".format(queryId) 18 | 19 | 20 | if __name__ == '__main__': 21 | send_request() 22 | 23 | 24 | -------------------------------------------------------------------------------- /CDB-Server/scripts/test_scripts/test_server.py: -------------------------------------------------------------------------------- 1 | import web 2 | import json 3 | 4 | urls = ( 5 | '/upload/', 'Upload', 6 | '/check/', 'Check', 7 | '/results/', 'Result' 8 | ) 9 | 10 | hits = {} 11 | columns = [] 12 | choices = [] 13 | options = None 14 | limit = 5 15 | 16 | class Upload: 17 | def POST(self): 18 | 19 | global hits, columns, choices, options, limit 20 | data_str = web.input().data 21 | # print data_str 22 | data = json.loads(data_str) 23 | options = data["options"] 24 | questions = data["questions"] 25 | 26 | 27 | hits.clear() 28 | for question in questions: 29 | hits[question["id"]] = 1 30 | if "columns" in question: 31 | columns = question["columns"] 32 | if "options" in question: 33 | choices = question["options"] 34 | 35 | 36 | results = { 37 | "code": 0, 38 | "status": 1, 39 | } 40 | return json.dumps(results) 41 | 42 | 43 | class Check: 44 | def GET(self): 45 | results = { 46 | "code": 0, 47 | "status": True, 48 | } 49 | return json.dumps(results) 50 | 51 | class Result: 52 | def GET(self): 53 | res = {} 54 | global options, limit 55 | for hit, label in hits.iteritems(): 56 | ans = {} 57 | ans["id"] = hit 58 | if options["task_type"] == "fill": 59 | qans = {} 60 | for col in columns: 61 | qans[col] = "col" 62 | ans["answer"] = qans 63 | elif options["task_type"] == "collect": 64 | pairs = [] 65 | for i in xrange(0, limit): 66 | entity = {} 67 | for attr in columns: 68 | entity[attr] = "col" 69 | pairs.append(entity) 70 | ans["answer"] = pairs 71 | elif options["task_type"] == "single_label" or options["task_type"] == "multi_label": 72 | qans = choices 73 | ans["answer"] = ["snow", "sunny"] 74 | else: 75 | ans["answer"] = 1 76 | res[hit] = ans 77 | 78 | results = { 79 | "code": 0, 80 | "data": res, 81 | } 82 | 83 | print json.dumps(results) 84 | return json.dumps(results) 85 | 86 | if __name__ == "__main__": 87 | app = web.application(urls, globals()) 88 | app.run() -------------------------------------------------------------------------------- /CDB-Server/start.sh: -------------------------------------------------------------------------------- 1 | nohup mvn exec:java -pl crowdscheduler -Dmaven.test.skip=true & 2 | -------------------------------------------------------------------------------- /CDB-WEB/README.md: -------------------------------------------------------------------------------- 1 | # CDB-WEB - An Web Service for CrowdDB 2 | 3 | ## Introduction 4 | 5 | This is a web service demo UI for CrowdDB. By using the web UI, you can create your own crowd-sql, check the processing progress, and collect the sql results in your own database. 6 | 7 | ## About The Service 8 | 9 | To manage your crowd-sql with this service, you should provide an database user with high privileges (including GRANT privileges) in service/app.py. Then the service would create a database user for every user registered who registered in the service with the username and password, and grant privileges for it. 10 | 11 | All the crowd-sql operations would be stored and managed in the database using the registered database user. 12 | 13 | ## Pre Requirements 14 | 15 | Node.js >= v5.7.1 16 | Python2 17 | pip 18 | 19 | 20 | ## Build and Run 21 | 22 | ``` bash 23 | # build web page of the service 24 | cd front 25 | npm install 26 | npm run build 27 | cd .. 28 | 29 | # copy the web page into the service folder 30 | cp front/dist/index.html service/templates/ 31 | cp front/dist/static/* service/static/ 32 | 33 | # install python dependencies 34 | apt-get install libmysqlclient-dev 35 | apt-get install python-mysqldb 36 | cd service 37 | pip install -r requirements.txt 38 | 39 | # configure the database with your own settings in app.py 40 | vim service/app.py 41 | 42 | # initialize database 43 | python model.py db init 44 | python model.py db upgrade 45 | 46 | # run demo 47 | python web.py 48 | 49 | ``` 50 | -------------------------------------------------------------------------------- /CDB-WEB/front/.babelrc: -------------------------------------------------------------------------------- 1 | { 2 | "presets": ["es2015", "stage-2"], 3 | "plugins": ["transform-runtime"], 4 | "comments": false 5 | } 6 | -------------------------------------------------------------------------------- /CDB-WEB/front/.editorconfig: -------------------------------------------------------------------------------- 1 | root = true 2 | 3 | [*] 4 | charset = utf-8 5 | indent_style = space 6 | indent_size = 2 7 | end_of_line = lf 8 | insert_final_newline = true 9 | trim_trailing_whitespace = true 10 | -------------------------------------------------------------------------------- /CDB-WEB/front/.gitignore: -------------------------------------------------------------------------------- 1 | .DS_Store 2 | node_modules/ 3 | dist/ 4 | npm-debug.log 5 | -------------------------------------------------------------------------------- /CDB-WEB/front/README.md: -------------------------------------------------------------------------------- 1 | # front 2 | 3 | > A Vue.js project for the front end of CrowdDB Web Service 4 | 5 | ## Build Setup 6 | 7 | ``` bash 8 | # install dependencies 9 | npm install 10 | 11 | # serve with hot reload at localhost:8080 12 | npm run dev 13 | 14 | # build for production with minification 15 | npm run build 16 | ``` 17 | -------------------------------------------------------------------------------- /CDB-WEB/front/build/build.js: -------------------------------------------------------------------------------- 1 | // https://github.com/shelljs/shelljs 2 | require('./check-versions')() 3 | require('shelljs/global') 4 | env.NODE_ENV = 'production' 5 | 6 | var path = require('path') 7 | var config = require('../config') 8 | var ora = require('ora') 9 | var webpack = require('webpack') 10 | var webpackConfig = require('./webpack.prod.conf') 11 | 12 | console.log( 13 | ' Tip:\n' + 14 | ' Built files are meant to be served over an HTTP server.\n' + 15 | ' Opening index.html over file:// won\'t work.\n' 16 | ) 17 | 18 | var spinner = ora('building for production...') 19 | spinner.start() 20 | 21 | var assetsPath = path.join(config.build.assetsRoot, config.build.assetsSubDirectory) 22 | rm('-rf', assetsPath) 23 | mkdir('-p', assetsPath) 24 | cp('-R', 'static/*', assetsPath) 25 | 26 | webpack(webpackConfig, function (err, stats) { 27 | spinner.stop() 28 | if (err) throw err 29 | process.stdout.write(stats.toString({ 30 | colors: true, 31 | modules: false, 32 | children: false, 33 | chunks: false, 34 | chunkModules: false 35 | }) + '\n') 36 | }) 37 | -------------------------------------------------------------------------------- /CDB-WEB/front/build/check-versions.js: -------------------------------------------------------------------------------- 1 | var semver = require('semver') 2 | var chalk = require('chalk') 3 | var packageConfig = require('../package.json') 4 | var exec = function (cmd) { 5 | return require('child_process') 6 | .execSync(cmd).toString().trim() 7 | } 8 | 9 | var versionRequirements = [ 10 | { 11 | name: 'node', 12 | currentVersion: semver.clean(process.version), 13 | versionRequirement: packageConfig.engines.node 14 | }, 15 | { 16 | name: 'npm', 17 | currentVersion: exec('npm --version'), 18 | versionRequirement: packageConfig.engines.npm 19 | } 20 | ] 21 | 22 | module.exports = function () { 23 | var warnings = [] 24 | for (var i = 0; i < versionRequirements.length; i++) { 25 | var mod = versionRequirements[i] 26 | if (!semver.satisfies(mod.currentVersion, mod.versionRequirement)) { 27 | warnings.push(mod.name + ': ' + 28 | chalk.red(mod.currentVersion) + ' should be ' + 29 | chalk.green(mod.versionRequirement) 30 | ) 31 | } 32 | } 33 | 34 | if (warnings.length) { 35 | console.log('') 36 | console.log(chalk.yellow('To use this template, you must update following to modules:')) 37 | console.log() 38 | for (var i = 0; i < warnings.length; i++) { 39 | var warning = warnings[i] 40 | console.log(' ' + warning) 41 | } 42 | console.log() 43 | process.exit(1) 44 | } 45 | } 46 | -------------------------------------------------------------------------------- /CDB-WEB/front/build/dev-client.js: -------------------------------------------------------------------------------- 1 | /* eslint-disable */ 2 | require('eventsource-polyfill') 3 | var hotClient = require('webpack-hot-middleware/client?noInfo=true&reload=true') 4 | 5 | hotClient.subscribe(function (event) { 6 | if (event.action === 'reload') { 7 | window.location.reload() 8 | } 9 | }) 10 | -------------------------------------------------------------------------------- /CDB-WEB/front/build/dev-server.js: -------------------------------------------------------------------------------- 1 | require('./check-versions')() 2 | var config = require('../config') 3 | if (!process.env.NODE_ENV) process.env.NODE_ENV = JSON.parse(config.dev.env.NODE_ENV) 4 | var path = require('path') 5 | var express = require('express') 6 | var webpack = require('webpack') 7 | var opn = require('opn') 8 | var proxyMiddleware = require('http-proxy-middleware') 9 | var webpackConfig = require('./webpack.dev.conf') 10 | 11 | // default port where dev server listens for incoming traffic 12 | var port = process.env.PORT || config.dev.port 13 | // Define HTTP proxies to your custom API backend 14 | // https://github.com/chimurai/http-proxy-middleware 15 | var proxyTable = config.dev.proxyTable 16 | 17 | var app = express() 18 | var compiler = webpack(webpackConfig) 19 | 20 | var devMiddleware = require('webpack-dev-middleware')(compiler, { 21 | publicPath: webpackConfig.output.publicPath, 22 | stats: { 23 | colors: true, 24 | chunks: false 25 | } 26 | }) 27 | 28 | var hotMiddleware = require('webpack-hot-middleware')(compiler) 29 | // force page reload when html-webpack-plugin template changes 30 | compiler.plugin('compilation', function (compilation) { 31 | compilation.plugin('html-webpack-plugin-after-emit', function (data, cb) { 32 | hotMiddleware.publish({ action: 'reload' }) 33 | cb() 34 | }) 35 | }) 36 | 37 | // proxy api requests 38 | Object.keys(proxyTable).forEach(function (context) { 39 | var options = proxyTable[context] 40 | if (typeof options === 'string') { 41 | options = { target: options } 42 | } 43 | app.use(proxyMiddleware(context, options)) 44 | }) 45 | 46 | // handle fallback for HTML5 history API 47 | app.use(require('connect-history-api-fallback')()) 48 | 49 | // serve webpack bundle output 50 | app.use(devMiddleware) 51 | 52 | // enable hot-reload and state-preserving 53 | // compilation error display 54 | app.use(hotMiddleware) 55 | 56 | // serve pure static assets 57 | var staticPath = path.posix.join(config.dev.assetsPublicPath, config.dev.assetsSubDirectory) 58 | app.use(staticPath, express.static('./static')) 59 | 60 | module.exports = app.listen(port, function (err) { 61 | if (err) { 62 | console.log(err) 63 | return 64 | } 65 | var uri = 'http://localhost:' + port 66 | console.log('Listening at ' + uri + '\n') 67 | 68 | // when env is testing, don't need open it 69 | if (process.env.NODE_ENV !== 'testing') { 70 | opn(uri) 71 | } 72 | }) 73 | -------------------------------------------------------------------------------- /CDB-WEB/front/build/utils.js: -------------------------------------------------------------------------------- 1 | var path = require('path') 2 | var config = require('../config') 3 | var ExtractTextPlugin = require('extract-text-webpack-plugin') 4 | 5 | exports.assetsPath = function (_path) { 6 | var assetsSubDirectory = process.env.NODE_ENV === 'production' 7 | ? config.build.assetsSubDirectory 8 | : config.dev.assetsSubDirectory 9 | return path.posix.join(assetsSubDirectory, _path) 10 | } 11 | 12 | exports.cssLoaders = function (options) { 13 | options = options || {} 14 | // generate loader string to be used with extract text plugin 15 | function generateLoaders (loaders) { 16 | var sourceLoader = loaders.map(function (loader) { 17 | var extraParamChar 18 | if (/\?/.test(loader)) { 19 | loader = loader.replace(/\?/, '-loader?') 20 | extraParamChar = '&' 21 | } else { 22 | loader = loader + '-loader' 23 | extraParamChar = '?' 24 | } 25 | return loader + (options.sourceMap ? extraParamChar + 'sourceMap' : '') 26 | }).join('!') 27 | 28 | // Extract CSS when that option is specified 29 | // (which is the case during production build) 30 | if (options.extract) { 31 | return ExtractTextPlugin.extract('vue-style-loader', sourceLoader) 32 | } else { 33 | return ['vue-style-loader', sourceLoader].join('!') 34 | } 35 | } 36 | 37 | // http://vuejs.github.io/vue-loader/en/configurations/extract-css.html 38 | return { 39 | css: generateLoaders(['css']), 40 | postcss: generateLoaders(['css']), 41 | less: generateLoaders(['css', 'less']), 42 | sass: generateLoaders(['css', 'sass?indentedSyntax']), 43 | scss: generateLoaders(['css', 'sass']), 44 | stylus: generateLoaders(['css', 'stylus']), 45 | styl: generateLoaders(['css', 'stylus']) 46 | } 47 | } 48 | 49 | // Generate loaders for standalone style files (outside of .vue) 50 | exports.styleLoaders = function (options) { 51 | var output = [] 52 | var loaders = exports.cssLoaders(options) 53 | for (var extension in loaders) { 54 | var loader = loaders[extension] 55 | output.push({ 56 | test: new RegExp('\\.' + extension + '$'), 57 | loader: loader 58 | }) 59 | } 60 | return output 61 | } 62 | -------------------------------------------------------------------------------- /CDB-WEB/front/build/webpack.base.conf.js: -------------------------------------------------------------------------------- 1 | var path = require('path') 2 | var config = require('../config') 3 | var utils = require('./utils') 4 | var projectRoot = path.resolve(__dirname, '../') 5 | 6 | var env = process.env.NODE_ENV 7 | // check env & config/index.js to decide weither to enable CSS Sourcemaps for the 8 | // various preprocessor loaders added to vue-loader at the end of this file 9 | var cssSourceMapDev = (env === 'development' && config.dev.cssSourceMap) 10 | var cssSourceMapProd = (env === 'production' && config.build.productionSourceMap) 11 | var useCssSourceMap = cssSourceMapDev || cssSourceMapProd 12 | 13 | module.exports = { 14 | entry: { 15 | app: './src/main.js' 16 | }, 17 | output: { 18 | path: config.build.assetsRoot, 19 | publicPath: process.env.NODE_ENV === 'production' ? config.build.assetsPublicPath : config.dev.assetsPublicPath, 20 | filename: '[name].js' 21 | }, 22 | resolve: { 23 | extensions: ['', '.js', '.vue'], 24 | fallback: [path.join(__dirname, '../node_modules')], 25 | alias: { 26 | 'vue$': 'vue/dist/vue.common.js', 27 | 'src': path.resolve(__dirname, '../src'), 28 | 'assets': path.resolve(__dirname, '../src/assets'), 29 | 'components': path.resolve(__dirname, '../src/components') 30 | } 31 | }, 32 | resolveLoader: { 33 | fallback: [path.join(__dirname, '../node_modules')] 34 | }, 35 | module: { 36 | loaders: [ 37 | { 38 | test: /\.vue$/, 39 | loader: 'vue' 40 | }, 41 | { 42 | test: /\.js$/, 43 | loader: 'babel', 44 | include: projectRoot, 45 | exclude: /node_modules/ 46 | }, 47 | { 48 | test: /\.json$/, 49 | loader: 'json' 50 | }, 51 | { 52 | test: /\.(png|jpe?g|gif|svg)(\?.*)?$/, 53 | loader: 'url', 54 | query: { 55 | limit: 10000, 56 | name: utils.assetsPath('img/[name].[hash:7].[ext]') 57 | } 58 | }, 59 | { 60 | test: /\.(woff2?|eot|ttf|otf)(\?.*)?$/, 61 | loader: 'url', 62 | query: { 63 | limit: 10000, 64 | name: utils.assetsPath('fonts/[name].[hash:7].[ext]') 65 | } 66 | } 67 | ] 68 | }, 69 | vue: { 70 | loaders: utils.cssLoaders({ sourceMap: useCssSourceMap }), 71 | postcss: [ 72 | require('autoprefixer')({ 73 | browsers: ['last 2 versions'] 74 | }) 75 | ] 76 | } 77 | } 78 | -------------------------------------------------------------------------------- /CDB-WEB/front/build/webpack.dev.conf.js: -------------------------------------------------------------------------------- 1 | var config = require('../config') 2 | var webpack = require('webpack') 3 | var merge = require('webpack-merge') 4 | var utils = require('./utils') 5 | var baseWebpackConfig = require('./webpack.base.conf') 6 | var HtmlWebpackPlugin = require('html-webpack-plugin') 7 | 8 | // add hot-reload related code to entry chunks 9 | Object.keys(baseWebpackConfig.entry).forEach(function (name) { 10 | baseWebpackConfig.entry[name] = ['./build/dev-client'].concat(baseWebpackConfig.entry[name]) 11 | }) 12 | 13 | module.exports = merge(baseWebpackConfig, { 14 | module: { 15 | loaders: utils.styleLoaders({ sourceMap: config.dev.cssSourceMap }) 16 | }, 17 | // eval-source-map is faster for development 18 | devtool: '#eval-source-map', 19 | plugins: [ 20 | new webpack.DefinePlugin({ 21 | 'process.env': config.dev.env 22 | }), 23 | // https://github.com/glenjamin/webpack-hot-middleware#installation--usage 24 | new webpack.optimize.OccurenceOrderPlugin(), 25 | new webpack.HotModuleReplacementPlugin(), 26 | new webpack.NoErrorsPlugin(), 27 | // https://github.com/ampedandwired/html-webpack-plugin 28 | new HtmlWebpackPlugin({ 29 | filename: 'index.html', 30 | template: 'index.html', 31 | inject: true 32 | }) 33 | ] 34 | }) 35 | -------------------------------------------------------------------------------- /CDB-WEB/front/build/webpack.prod.conf.js: -------------------------------------------------------------------------------- 1 | var path = require('path') 2 | var config = require('../config') 3 | var utils = require('./utils') 4 | var webpack = require('webpack') 5 | var merge = require('webpack-merge') 6 | var baseWebpackConfig = require('./webpack.base.conf') 7 | var ExtractTextPlugin = require('extract-text-webpack-plugin') 8 | var HtmlWebpackPlugin = require('html-webpack-plugin') 9 | var env = config.build.env 10 | 11 | var webpackConfig = merge(baseWebpackConfig, { 12 | module: { 13 | loaders: utils.styleLoaders({ sourceMap: config.build.productionSourceMap, extract: true }) 14 | }, 15 | devtool: config.build.productionSourceMap ? '#source-map' : false, 16 | output: { 17 | path: config.build.assetsRoot, 18 | filename: utils.assetsPath('js/[name].[chunkhash].js'), 19 | chunkFilename: utils.assetsPath('js/[id].[chunkhash].js') 20 | }, 21 | vue: { 22 | loaders: utils.cssLoaders({ 23 | sourceMap: config.build.productionSourceMap, 24 | extract: true 25 | }) 26 | }, 27 | plugins: [ 28 | // http://vuejs.github.io/vue-loader/en/workflow/production.html 29 | new webpack.DefinePlugin({ 30 | 'process.env': env 31 | }), 32 | new webpack.optimize.UglifyJsPlugin({ 33 | compress: { 34 | warnings: false 35 | } 36 | }), 37 | new webpack.optimize.OccurrenceOrderPlugin(), 38 | // extract css into its own file 39 | new ExtractTextPlugin(utils.assetsPath('css/[name].[contenthash].css')), 40 | // generate dist index.html with correct asset hash for caching. 41 | // you can customize output by editing /index.html 42 | // see https://github.com/ampedandwired/html-webpack-plugin 43 | new HtmlWebpackPlugin({ 44 | filename: config.build.index, 45 | template: 'index.html', 46 | inject: true, 47 | minify: { 48 | removeComments: true, 49 | collapseWhitespace: true, 50 | removeAttributeQuotes: true 51 | // more options: 52 | // https://github.com/kangax/html-minifier#options-quick-reference 53 | }, 54 | // necessary to consistently work with multiple chunks via CommonsChunkPlugin 55 | chunksSortMode: 'dependency' 56 | }), 57 | // split vendor js into its own file 58 | new webpack.optimize.CommonsChunkPlugin({ 59 | name: 'vendor', 60 | minChunks: function (module, count) { 61 | // any required modules inside node_modules are extracted to vendor 62 | return ( 63 | module.resource && 64 | /\.js$/.test(module.resource) && 65 | module.resource.indexOf( 66 | path.join(__dirname, '../node_modules') 67 | ) === 0 68 | ) 69 | } 70 | }), 71 | // extract webpack runtime and module manifest to its own file in order to 72 | // prevent vendor hash from being updated whenever app bundle is updated 73 | new webpack.optimize.CommonsChunkPlugin({ 74 | name: 'manifest', 75 | chunks: ['vendor'] 76 | }) 77 | ] 78 | }) 79 | 80 | if (config.build.productionGzip) { 81 | var CompressionWebpackPlugin = require('compression-webpack-plugin') 82 | 83 | webpackConfig.plugins.push( 84 | new CompressionWebpackPlugin({ 85 | asset: '[path].gz[query]', 86 | algorithm: 'gzip', 87 | test: new RegExp( 88 | '\\.(' + 89 | config.build.productionGzipExtensions.join('|') + 90 | ')$' 91 | ), 92 | threshold: 10240, 93 | minRatio: 0.8 94 | }) 95 | ) 96 | } 97 | 98 | module.exports = webpackConfig 99 | -------------------------------------------------------------------------------- /CDB-WEB/front/config/dev.env.js: -------------------------------------------------------------------------------- 1 | var merge = require('webpack-merge') 2 | var prodEnv = require('./prod.env') 3 | 4 | module.exports = merge(prodEnv, { 5 | NODE_ENV: '"development"' 6 | }) 7 | -------------------------------------------------------------------------------- /CDB-WEB/front/config/index.js: -------------------------------------------------------------------------------- 1 | // see http://vuejs-templates.github.io/webpack for documentation. 2 | var path = require('path') 3 | 4 | module.exports = { 5 | build: { 6 | env: require('./prod.env'), 7 | index: path.resolve(__dirname, '../dist/index.html'), 8 | assetsRoot: path.resolve(__dirname, '../dist'), 9 | assetsSubDirectory: 'static', 10 | assetsPublicPath: '/', 11 | productionSourceMap: true, 12 | // Gzip off by default as many popular static hosts such as 13 | // Surge or Netlify already gzip all static assets for you. 14 | // Before setting to `true`, make sure to: 15 | // npm install --save-dev compression-webpack-plugin 16 | productionGzip: false, 17 | productionGzipExtensions: ['js', 'css'] 18 | }, 19 | dev: { 20 | env: require('./dev.env'), 21 | port: 8080, 22 | assetsSubDirectory: 'static', 23 | assetsPublicPath: '/', 24 | proxyTable: {}, 25 | // CSS Sourcemaps off by default because relative paths are "buggy" 26 | // with this option, according to the CSS-Loader README 27 | // (https://github.com/webpack/css-loader#sourcemaps) 28 | // In our experience, they generally work as expected, 29 | // just be aware of this issue when enabling this option. 30 | cssSourceMap: false 31 | } 32 | } 33 | -------------------------------------------------------------------------------- /CDB-WEB/front/config/prod.env.js: -------------------------------------------------------------------------------- 1 | module.exports = { 2 | NODE_ENV: '"production"' 3 | } 4 | -------------------------------------------------------------------------------- /CDB-WEB/front/index.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | CrowdDB 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 |
21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | -------------------------------------------------------------------------------- /CDB-WEB/front/package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "front", 3 | "version": "1.0.0", 4 | "description": "A Vue.js project", 5 | "author": "lihaoda ", 6 | "private": true, 7 | "scripts": { 8 | "dev": "node build/dev-server.js", 9 | "build": "node build/build.js" 10 | }, 11 | "dependencies": { 12 | "js-cookie": "^2.1.3", 13 | "vue": "^2.1.0", 14 | "vue-resource": "^1.0.3", 15 | "vue-router": "^2.1.1", 16 | "vuex": "^2.0.0" 17 | }, 18 | "devDependencies": { 19 | "autoprefixer": "^6.4.0", 20 | "babel-core": "^6.0.0", 21 | "babel-loader": "^6.0.0", 22 | "babel-plugin-transform-runtime": "^6.0.0", 23 | "babel-preset-es2015": "^6.0.0", 24 | "babel-preset-stage-2": "^6.0.0", 25 | "babel-register": "^6.0.0", 26 | "chalk": "^1.1.3", 27 | "connect-history-api-fallback": "^1.1.0", 28 | "css-loader": "^0.25.0", 29 | "eventsource-polyfill": "^0.9.6", 30 | "express": "^4.13.3", 31 | "extract-text-webpack-plugin": "^1.0.1", 32 | "file-loader": "^0.9.0", 33 | "function-bind": "^1.0.2", 34 | "html-webpack-plugin": "^2.8.1", 35 | "http-proxy-middleware": "^0.17.2", 36 | "json-loader": "^0.5.4", 37 | "semver": "^5.3.0", 38 | "opn": "^4.0.2", 39 | "ora": "^0.3.0", 40 | "shelljs": "^0.7.4", 41 | "url-loader": "^0.5.7", 42 | "vue-loader": "^10.0.0", 43 | "vue-style-loader": "^1.0.0", 44 | "vue-template-compiler": "^2.1.0", 45 | "webpack": "^1.13.2", 46 | "webpack-dev-middleware": "^1.8.3", 47 | "webpack-hot-middleware": "^2.12.2", 48 | "webpack-merge": "^0.14.1" 49 | }, 50 | "engines": { 51 | "node": ">= 4.0.0", 52 | "npm": ">= 3.0.0" 53 | } 54 | } 55 | -------------------------------------------------------------------------------- /CDB-WEB/front/src/App.vue: -------------------------------------------------------------------------------- 1 | 12 | 13 | 39 | 40 | 61 | -------------------------------------------------------------------------------- /CDB-WEB/front/src/assets/login.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TsinghuaDatabaseGroup/CDB/6d949d89247598d8785d59d25d0557726d4e652f/CDB-WEB/front/src/assets/login.jpg -------------------------------------------------------------------------------- /CDB-WEB/front/src/assets/logo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TsinghuaDatabaseGroup/CDB/6d949d89247598d8785d59d25d0557726d4e652f/CDB-WEB/front/src/assets/logo.png -------------------------------------------------------------------------------- /CDB-WEB/front/src/components/Hello.vue: -------------------------------------------------------------------------------- 1 | 28 | 29 | 30 | 49 | 50 | 87 | -------------------------------------------------------------------------------- /CDB-WEB/front/src/main.js: -------------------------------------------------------------------------------- 1 | // The Vue build version to load with the `import` command 2 | // (runtime-only or standalone) has been set in webpack.base.conf with an alias. 3 | import Vue from 'vue' 4 | import VueResource from 'vue-resource' 5 | 6 | import store from './store/index' 7 | import * as types from './store/mutation_types' 8 | import * as actions from './store/actions' 9 | import Cookies from 'js-cookie' 10 | import App from './App' 11 | 12 | Vue.use(VueResource); 13 | 14 | Vue.http.options.xhr = {withCredentials: true}; 15 | 16 | Vue.http.interceptors.push( 17 | (req, next) => { 18 | req.headers.map['Authorization'] = ['JWT ' + store.state.token]; 19 | req.headers.map['Accept'] = ['application/json']; 20 | console.log(req); 21 | next((res) => { 22 | console.log(res); 23 | if("resetAuthToken" in res.data && res.data.resetAuthToken) { 24 | store.commit(types.SET_TOKEN, ""); 25 | } 26 | return res; 27 | }) 28 | } 29 | ); 30 | 31 | Vue.filter('len_limit20', function (value) { 32 | if(value.substring) { 33 | return value.substring(0, 20); 34 | } else { 35 | return value; 36 | } 37 | }); 38 | 39 | var crowddb_token = Cookies.get("crowddb_token"); 40 | if(crowddb_token) { 41 | store.commit(types.SET_TOKEN, crowddb_token); 42 | actions.testToken(store, function(data) { 43 | new Vue({ 44 | el: '#app', 45 | template: '', 46 | components: { App } 47 | }); 48 | }, function(data) { 49 | new Vue({ 50 | el: '#app', 51 | template: '', 52 | components: { App } 53 | }); 54 | }); 55 | } else { 56 | new Vue({ 57 | el: '#app', 58 | template: '', 59 | components: { App } 60 | }); 61 | } 62 | -------------------------------------------------------------------------------- /CDB-WEB/front/src/store/actions.js: -------------------------------------------------------------------------------- 1 | /** 2 | * Created by lihaoda. 3 | */ 4 | 5 | import Vue from 'vue' 6 | import * as types from './mutation_types' 7 | 8 | export const testToken = (store, succall, errcall) => { 9 | Vue.http.post(store.state.apiUrl + '/api/test_auth', {}).then(function(res) { 10 | if(res.data.success) { 11 | succall(res.data); 12 | } else { 13 | errcall(res); 14 | } 15 | }, errcall) 16 | }; 17 | 18 | export const register = (store, obj, succall, errcall) => { 19 | Vue.http.post(store.state.apiUrl + '/api/register', obj).then(function(res) { 20 | if(res.data.success) { 21 | succall(res.data); 22 | } else { 23 | errcall(res); 24 | } 25 | }, errcall) 26 | }; 27 | 28 | export const login = (store, obj, succall, errcall) => { 29 | Vue.http.post(store.state.apiUrl + '/api/auth', obj).then(function(res) { 30 | if(res.data.access_token) { 31 | store.commit(types.SET_TOKEN, res.data.access_token); 32 | succall(res.data); 33 | } else { 34 | errcall(res); 35 | } 36 | }, errcall) 37 | }; 38 | 39 | export const runNormalSql = (store, obj, succall, errcall) => { 40 | Vue.http.post(store.state.apiUrl + '/api/run_simple_sql', obj).then(function(res) { 41 | if(res.data.success) { 42 | succall(res.data); 43 | } else { 44 | errcall(res); 45 | } 46 | }, errcall) 47 | }; 48 | 49 | export const runCrowdSql = (store, obj, succall, errcall) => { 50 | Vue.http.post(store.state.apiUrl + '/api/run_crowd_sql', obj).then(function(res) { 51 | if(res.data.success) { 52 | succall(res.data); 53 | } else { 54 | errcall(res); 55 | } 56 | }, errcall) 57 | }; 58 | 59 | export const removeCrowdSql = (store, obj, succall, errcall) => { 60 | Vue.http.post(store.state.apiUrl + '/api/delete_crowd_sql', obj).then(function(res) { 61 | if(res.data.success) { 62 | succall(res.data); 63 | } else { 64 | errcall(res); 65 | } 66 | }, errcall) 67 | }; 68 | 69 | export const get_crowd_sqls = (store, obj, succall, errcall) => { 70 | Vue.http.post(store.state.apiUrl + '/api/get_all_crowd_sql', obj).then(function(res) { 71 | if(res.data.success) { 72 | succall(res.data); 73 | } else { 74 | errcall(res); 75 | } 76 | }, errcall) 77 | }; 78 | 79 | 80 | 81 | 82 | 83 | 84 | 85 | -------------------------------------------------------------------------------- /CDB-WEB/front/src/store/index.js: -------------------------------------------------------------------------------- 1 | /** 2 | * Created by lihaoda. 3 | */ 4 | 5 | import Vue from 'vue' 6 | import Vuex from 'vuex' 7 | import * as types from './mutation_types' 8 | import Cookies from 'js-cookie' 9 | 10 | Vue.use(Vuex); 11 | const debug = process.env.NODE_ENV !== 'production'; 12 | 13 | const state = { 14 | token: "", 15 | apiUrl: "" 16 | }; 17 | 18 | const mutations = { 19 | [types.SET_TOKEN] (state, t) { 20 | Cookies.set('crowddb_token', t); 21 | state.token = t; 22 | }, 23 | }; 24 | 25 | export default new Vuex.Store({ 26 | state, 27 | mutations, 28 | strict: debug 29 | ? [/*createLogger()*/] 30 | : [] 31 | }); 32 | -------------------------------------------------------------------------------- /CDB-WEB/front/src/store/mutation_types.js: -------------------------------------------------------------------------------- 1 | /** 2 | * Created by lihaoda. 3 | */ 4 | 5 | export const SET_TOKEN = 'SET_TOKEN'; 6 | -------------------------------------------------------------------------------- /CDB-WEB/front/static/.gitkeep: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TsinghuaDatabaseGroup/CDB/6d949d89247598d8785d59d25d0557726d4e652f/CDB-WEB/front/static/.gitkeep -------------------------------------------------------------------------------- /CDB-WEB/service/README.md: -------------------------------------------------------------------------------- 1 | # service 2 | 3 | > A Flask project of CrowdDB Web Service 4 | 5 | ## Build Setup 6 | 7 | ``` bash 8 | # install dependencies 9 | apt-get install libmysqlclient-dev 10 | apt-get install python-mysqldb 11 | pip install -r requirements.txt 12 | 13 | # configure the database with your own settings in app.py 14 | vim app.py 15 | 16 | # initialize database 17 | python model.py db init 18 | python model.py db upgrade 19 | 20 | # run demo 21 | python web.py 22 | 23 | ``` 24 | -------------------------------------------------------------------------------- /CDB-WEB/service/app.py: -------------------------------------------------------------------------------- 1 | # -*- coding:utf-8 -*- 2 | 3 | from flask import Flask 4 | from flask_cors import CORS 5 | from datetime import datetime, timedelta 6 | 7 | ## MySQL 8 | DB_USERNAME = '???????????' 9 | DB_PASSWORD = '??????????????????????????' 10 | DB_HOST = 'localhost' 11 | DB_PORT = '3306' 12 | DB_DBNAME = 'crowddb_user_meta' 13 | CROWD_META_DB = 'crowddb_meta' 14 | 15 | db_conn_str = 'mysql+mysqldb://%s:%s@%s:%s/%s?charset=utf8' % (DB_USERNAME, DB_PASSWORD, DB_HOST, DB_PORT, DB_DBNAME) 16 | crowd_conn_str = 'mysql+mysqldb://%s:%s@%s:%s/%s?charset=utf8' % (DB_USERNAME, DB_PASSWORD, DB_HOST, DB_PORT, CROWD_META_DB) 17 | user_db_template = 'mysql+mysqldb://%s:%s@%s:%s/%s?charset=utf8' % ("%s", "%s", DB_HOST, DB_PORT, "%s") 18 | 19 | CROWD_HOST = "127.0.0.1" 20 | CROWD_PORT = 1234 21 | 22 | ### Server Settings 23 | 24 | app = Flask(__name__) 25 | 26 | app.config['SECRET_KEY'] = 'crowd_db_key_234rSd2#432s@234' 27 | app.config['JWT_EXPIRATION_DELTA'] = timedelta(days=1) 28 | app.config['JWT_AUTH_URL_RULE'] = None 29 | app.config['JWT_AUTH_EMAIL_KEY'] = "email" 30 | app.config['JWT_AUTH_URL_RULE'] = "/api/auth" 31 | #app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///app.db' 32 | app.config['SQLALCHEMY_DATABASE_URI'] = db_conn_str 33 | app.config['TEMP_DIR'] = 'tmp' 34 | app.config['UPLOAD_FOLDER'] = "./upload/" 35 | 36 | CORS(app) 37 | -------------------------------------------------------------------------------- /CDB-WEB/service/error.py: -------------------------------------------------------------------------------- 1 | errors = { 2 | 'JSON_PARSE': (1001, 'JSON parse error'), 3 | 'ARGUMENTS': (1002, 'Arguments error'), 4 | 'VALIDATE_CODE_EXPIRED': (2001, 'Validate code expired'), 5 | 'VALIDATE_CODE_WRONG': (2002, 'Validate code is wrong'), 6 | 'USER_EXISTS': (2003, 'Username exists'), 7 | 'USERNAME_PASSWORD_WRONG': (2005, 'Username or password is wrong'), 8 | 'NO_SUCH_USER': (2006, 'No such user'), 9 | 'PASSWORD_WRONG': (2007, 'Password is wrong'), 10 | 'CHECK ERROR': (3001, 'No result fits qurey'), 11 | 'PAGE OVERFLOW': (3002, 'Page is not valid'), 12 | 'LABEL ERROR': (3003, 'some label checked is not included'), 13 | 'NO_SUCH_CITY': (4001, 'city not found'), 14 | 'NO_SUCH_ATTRACTION': (4002, 'attraction not found'), 15 | 'NO_SUCH_COUNTRY': (4003, 'country not found'), 16 | 'NO_IMG_UPLOADED': (5001, 'no image file is uploaded'), 17 | 'NOT_VALID_IMAGE': (5002, 'the uploaded file is not image file'), 18 | 'SQL_ERROR': (6001, 'SQL execution error'), 19 | 'PERMISSION DENIED': (9000, 'Not authenticated to access this resource') 20 | } 21 | 22 | 23 | def error(key, msg=None, remark=None): 24 | if key in errors: 25 | e = errors[key] 26 | if not msg: 27 | return {'succeed':False, 'errno': e[0], 'errmsg': e[1], 'remark':remark} 28 | else: 29 | return {'succeed':False, 'errno': e[0], 'errmsg': msg, 'remark':remark} 30 | else: 31 | if not msg: 32 | msg = "Unknown" 33 | return { 'succeed':False, 'errno': None, 'errmsg': msg, 'remark':remark} 34 | 35 | def success(**kwargs): 36 | res = {'success': 1} 37 | if kwargs: 38 | res.update(kwargs) 39 | return res 40 | 41 | def maybe(res, key, **kwargs): 42 | if res: 43 | return success(**kwargs) 44 | else: 45 | return error(key) 46 | -------------------------------------------------------------------------------- /CDB-WEB/service/model.py: -------------------------------------------------------------------------------- 1 | # -*- coding:utf-8 -*- 2 | 3 | # -*- coding:utf-8 -*- 4 | 5 | from app import app 6 | 7 | from flask_sqlalchemy import SQLAlchemy 8 | from flask_script import Manager 9 | from flask_migrate import Migrate, MigrateCommand 10 | 11 | from sqlalchemy import * 12 | from sqlalchemy.orm import relationship, backref 13 | 14 | db = SQLAlchemy(app) 15 | migrate = Migrate(app, db) 16 | 17 | manager = Manager(app) 18 | manager.add_command('db', MigrateCommand) 19 | 20 | Base = db.Model 21 | 22 | class User(Base): 23 | __tablename__ = 'users' 24 | 25 | id = Column(Integer, primary_key=True) 26 | username = Column(String(128), unique=True) 27 | password = Column(String(256)) 28 | 29 | dbname = Column(String(128)) 30 | dbstring = Column(String(256)) 31 | 32 | def to_json(self): 33 | return { 34 | "id":self.id, 35 | "username":self.username 36 | } 37 | 38 | if __name__ == '__main__': 39 | #print dir(db) 40 | manager.run() 41 | -------------------------------------------------------------------------------- /CDB-WEB/service/remove_user.py: -------------------------------------------------------------------------------- 1 | # -*- coding:utf-8 -*- 2 | 3 | from dbcontrol import * 4 | from model import db, User 5 | import sys 6 | 7 | def remove_user(username): 8 | user = User.query.filter_by(username=username).first() 9 | if user: 10 | remove_user_env(user.username, user.dbname) 11 | db.session.delete(user) 12 | db.session.commit() 13 | else: 14 | raise Exception("User "+ username+ " not exists!") 15 | 16 | if __name__ == "__main__": 17 | print "Remove", sys.argv[1], "..." 18 | remove_user(sys.argv[1]) 19 | -------------------------------------------------------------------------------- /CDB-WEB/service/requirements.txt: -------------------------------------------------------------------------------- 1 | Werkzeug==0.11.5 2 | Flask==0.10.1 3 | Flask_Cors==2.1.2 4 | Flask_Script==2.0.5 5 | alembic==0.8.6 6 | validators==0.10.1 7 | pymongo==3.3.0 8 | Flask_JWT==0.3.2 9 | Flask_SQLAlchemy==2.1 10 | Flask_Migrate==1.8.0 11 | SQLAlchemy==1.0.13 12 | pytz==2016.6.1 13 | mysql-python 14 | -------------------------------------------------------------------------------- /CDB-WEB/service/static/css/app.8d53a3b4be94e580d5043f5e3d68dd4f.css: -------------------------------------------------------------------------------- 1 | #app{font-family:Avenir,Helvetica,Arial,sans-serif;-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;text-align:center;color:#2c3e50;margin-top:0}.clickable:hover{cursor:pointer}.clearfix:after{content:".";display:block;height:0;visibility:hidden;clear:both}h1[data-v-37209930],h2[data-v-37209930]{font-weight:400}ul[data-v-37209930]{list-style-type:none;padding:0}li[data-v-37209930]{display:inline-block;margin:0 10px}a[data-v-37209930]{color:#42b983}.show-on-hover[data-v-74110ab0]{display:none}tr+tr.show-on-hover[data-v-74110ab0]:hover,tr:hover+tr.show-on-hover[data-v-74110ab0]{display:table-row}.scroll-table-header[data-v-74110ab0]{overflow-y:hidden;overflow-x:hidden;background:#fff;position:relative;padding-right:19px;height:30px;z-index:1000}.scroll-table[data-v-74110ab0]{z-index:500;overflow-y:scroll;overflow-x:scroll;max-height:500px;margin-top:-30px}.scroll-table-small[data-v-74110ab0]{overflow-y:auto;overflow-x:hidden;max-height:340px;margin-top:-22px}.show-table[data-v-74110ab0]{margin:auto}.show-table tr[data-v-74110ab0]{padding:5px;border:1px solid}.show-table td[data-v-74110ab0]{padding:4px 5px;border:1px solid;min-width:30px}.sql-input[data-v-74110ab0]{margin-top:40px;width:95%;height:100%;font-family:Arial;font-size:medium}.show[data-v-74110ab0]{margin:auto;display:block;width:1000px;height:500px}.big-page[data-v-74110ab0]{width:100%;height:100%}.table-list[data-v-74110ab0]{max-height:600px;overflow-y:auto;overflow-x:hidden}.sql-list[data-v-74110ab0]{margin-left:-1px;border:2px;border:solid;border-color:brown;float:left;height:99%;width:15%;overflow:auto}.main-block[data-v-74110ab0]{float:left;height:100%;width:65%}.sql-block[data-v-74110ab0]{margin:10px;height:20%;width:80%}.meta-info[data-v-74110ab0]{height:25px;margin:5px;width:95%}.table-block[data-v-74110ab0]{margin:10px}h1[data-v-74110ab0],h2[data-v-74110ab0]{font-weight:400} 2 | /*# sourceMappingURL=app.8d53a3b4be94e580d5043f5e3d68dd4f.css.map*/ -------------------------------------------------------------------------------- /CDB-WEB/service/static/css/app.8d53a3b4be94e580d5043f5e3d68dd4f.css.map: -------------------------------------------------------------------------------- 1 | {"version":3,"sources":["webpack:///webpack:///src/App.vue","webpack:///webpack:///src/components/Hello.vue","webpack:///webpack:///src/components/CrowdShow.vue"],"names":[],"mappings":"AACA,KACE,8CACA,mCACA,kCACA,kBACA,cACA,YAAgB,CAElB,iBACE,cAAgB,CAElB,gBACE,YACA,cACA,SACA,kBACA,UAAY,CChBd,wCACE,eAAoB,CAEtB,oBACE,qBACA,SAAW,CAEb,oBACE,qBACA,aAAe,CAEjB,mBACE,aAAe,CCZjB,gCACE,YAAc,CAKhB,sFACE,iBAAmB,CAErB,sCACE,kBACA,kBACA,gBACA,kBACA,mBACA,YACA,YAAc,CAIhB,+BACE,YACA,kBACA,kBACA,iBACA,gBAAkB,CAEpB,qCACE,gBACA,kBACA,iBACA,gBAAkB,CAEpB,6BACE,WAAW,CAEb,gCACE,YACA,gBAAkB,CAEpB,gCACE,gBACA,iBACA,cAAgB,CAElB,4BACE,gBACA,UACA,YACA,kBACA,gBAAkB,CAEpB,uBACE,YACA,cACA,aACA,YAAc,CAEhB,2BACE,WACA,WAAa,CAEf,6BACE,iBACA,gBACA,iBAAmB,CAErB,2BACE,iBACA,WACA,aACA,mBACA,WACA,WACA,UACA,aAAe,CAEjB,6BACE,WACA,YACA,SAAU,CAEZ,4BACE,YACA,WACA,SAAW,CAEb,4BACE,YACA,WACA,SAAW,CAEb,8BACE,WAAa,CAEf,wCACE,eAAoB","file":"static/css/app.8d53a3b4be94e580d5043f5e3d68dd4f.css","sourcesContent":["\n#app {\n font-family: 'Avenir', Helvetica, Arial, sans-serif;\n -webkit-font-smoothing: antialiased;\n -moz-osx-font-smoothing: grayscale;\n text-align: center;\n color: #2c3e50;\n margin-top: 0px;\n}\n.clickable:hover {\n cursor: pointer;\n}\n.clearfix:after {\n content: \".\";\n display: block;\n height: 0;\n visibility: hidden;\n clear: both;\n}\n\n\n\n// WEBPACK FOOTER //\n// webpack:///src/App.vue","\nh1[data-v-37209930], h2[data-v-37209930] {\n font-weight: normal;\n}\nul[data-v-37209930] {\n list-style-type: none;\n padding: 0;\n}\nli[data-v-37209930] {\n display: inline-block;\n margin: 0 10px;\n}\na[data-v-37209930] {\n color: #42b983;\n}\n\n\n\n// WEBPACK FOOTER //\n// webpack:///src/components/Hello.vue","\n.show-on-hover[data-v-74110ab0] {\n display: none;\n}\ntr + tr.show-on-hover[data-v-74110ab0]:hover {\n display: table-row;\n}\ntr:hover + tr.show-on-hover[data-v-74110ab0] {\n display: table-row;\n}\n.scroll-table-header[data-v-74110ab0] {\n overflow-y: hidden;\n overflow-x: hidden;\n background:rgba(255,255,255,1);\n position:relative;\n padding-right: 19px;\n height: 30px;\n z-index: 1000;\n}\n.sub-category-container[data-v-74110ab0]{\n}\n.scroll-table[data-v-74110ab0] {\n z-index: 500;\n overflow-y: scroll;\n overflow-x: scroll;\n max-height: 500px;\n margin-top: -30px;\n}\n.scroll-table-small[data-v-74110ab0] {\n overflow-y: auto;\n overflow-x: hidden;\n max-height: 340px;\n margin-top: -22px;\n}\n.show-table[data-v-74110ab0] {\n margin:auto\n}\n.show-table tr[data-v-74110ab0] {\n padding:5px;\n border: solid 1px;\n}\n.show-table td[data-v-74110ab0] {\n padding: 4px 5px;\n border: solid 1px;\n min-width: 30px;\n}\n.sql-input[data-v-74110ab0] {\n margin-top: 40px;\n width: 95%;\n height: 100%;\n font-family: Arial;\n font-size: medium;\n}\n.show[data-v-74110ab0] {\n margin:auto;\n display: block;\n width: 1000px;\n height: 500px;\n}\n.big-page[data-v-74110ab0] {\n width: 100%;\n height: 100%;\n}\n.table-list[data-v-74110ab0] {\n max-height: 600px;\n overflow-y: auto;\n overflow-x: hidden;\n}\n.sql-list[data-v-74110ab0] {\n margin-left: -1px;\n border: 2px;\n border: solid;\n border-color: brown;\n float: left;\n height: 99%;\n width: 15%;\n overflow: auto;\n}\n.main-block[data-v-74110ab0] {\n float: left;\n height: 100%;\n width: 65%\n}\n.sql-block[data-v-74110ab0] {\n margin: 10px;\n height: 20%;\n width: 80%;\n}\n.meta-info[data-v-74110ab0] {\n height: 25px;\n margin: 5px 5px;\n width: 95%;\n}\n.table-block[data-v-74110ab0] {\n margin: 10px;\n}\nh1[data-v-74110ab0], h2[data-v-74110ab0] {\n font-weight: normal;\n}\n\n\n\n// WEBPACK FOOTER //\n// webpack:///src/components/CrowdShow.vue"],"sourceRoot":""} -------------------------------------------------------------------------------- /CDB-WEB/service/static/img/login.abdb9ad.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TsinghuaDatabaseGroup/CDB/6d949d89247598d8785d59d25d0557726d4e652f/CDB-WEB/service/static/img/login.abdb9ad.jpg -------------------------------------------------------------------------------- /CDB-WEB/service/static/js/manifest.dbb9dc97ad79361bbc8e.js: -------------------------------------------------------------------------------- 1 | !function(e){function t(n){if(r[n])return r[n].exports;var a=r[n]={exports:{},id:n,loaded:!1};return e[n].call(a.exports,a,a.exports,t),a.loaded=!0,a.exports}var n=window.webpackJsonp;window.webpackJsonp=function(o,p){for(var s,c,f=0,l=[];fCrowdDB
-------------------------------------------------------------------------------- /CDB-WEB/service/upload/.gitignore: -------------------------------------------------------------------------------- 1 | ./* 2 | 3 | -------------------------------------------------------------------------------- /CDB-WEB/service/util.py: -------------------------------------------------------------------------------- 1 | # -*- coding:utf-8 -*- 2 | import json 3 | from functools import wraps 4 | import hashlib 5 | 6 | from flask import jsonify, request 7 | from bson import ObjectId 8 | 9 | from datetime import datetime, timedelta 10 | #import pytz 11 | import time 12 | 13 | from math import radians, cos, sin, asin, sqrt 14 | import math 15 | 16 | import urllib2 17 | import re 18 | 19 | import imghdr 20 | import StringIO 21 | 22 | from checks import * 23 | from error import error, success 24 | 25 | def password_hash(string): 26 | return hashlib.md5(string).hexdigest() 27 | 28 | 29 | def datetime_round2day(dt): 30 | return dt - timedelta(hours=dt.hour, minutes=dt.minute, seconds=dt.second) 31 | 32 | def datetime2timestamp(dt): 33 | return int(time.mktime(dt.timetuple())) 34 | 35 | def timestamp2datetime(stamp, timezone_str="UTC"): 36 | tzinfo = pytz.timezone(timezone_str) 37 | return datetime.fromtimestamp(stamp).replace(tzinfo=pytz.utc).astimezone(tzinfo) 38 | 39 | def get_current_datetime(): 40 | return datetime.utcnow().replace(tzinfo=pytz.utc) 41 | 42 | def get_current_timestamp(): 43 | return datetime2timestamp(get_current_datetime()) 44 | 45 | def image_type(content): 46 | return imghdr.what(StringIO.StringIO(content)) 47 | 48 | class json_api(object): 49 | 50 | checkers = None 51 | 52 | def __init__(self, **checkers): 53 | self.checkers = checkers 54 | 55 | def __call__(self, func): 56 | @wraps(func) 57 | def wrapped(*args, **kwargs): 58 | if request.method != 'GET': 59 | j = None 60 | try: 61 | j = json.loads(request.data) 62 | except ValueError, e: 63 | return jsonify(error('JSON_PARSE')) 64 | errs = [] 65 | if not all([check_one_field( 66 | j, k, v, errs) for k, v in self.checkers.iteritems()]): 67 | return jsonify(error('ARGUMENTS', remark=errs)) 68 | kwargs.update( 69 | {k: j[k] for k in self.checkers if k in j}) 70 | return jsonify(func(*args, **kwargs)) 71 | return wrapped 72 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # CDB 2 | 3 | CDB is a crowd-powered database system that supports crowd-based query optimizations with focus on join and selection. 4 | CDB has fundamental differences from existing systems. First, CDB employs a graph-based query model that provides 5 | more fine-grained query optimization. Second, CDB adopts a unified framework to perform the multi-goal optimization based on the graph 6 | model. We have implemented our system and deployed it on Amazon Mechanical Turk, CrowdFlower and ChinaCrowd. 7 | 8 | ## Framework Description 9 | CDB is an intergated system with CDB-UI, CDB-Server and CDB-Assigment. End users will upload tables, submit queries and download results by CDB-UI. The 10 | query request will be submited to CDB-Server, and CDB-Server will parse sql, build model and execute every operator one by one. Those questions need to 11 | answer will be assigned to CDB-Platfrom, which will make up real questions and publish them into crowdsourcing platforms with platform. 12 | 13 | #### CDB-WEB 14 | 15 | CDB-WEB is a demo UI based on a web service for CrowdDB. By using the web UI, you can create your crowd-sql, check the processing progress, and collect the sql results in your own database. 16 | 17 | #### CDB-Server 18 | 19 | CDB-Server is the kernel component of CDB database. It provides two types of query model, one is straight-forward model and another is graph model. 20 | For both of them, it firstly parses the inputed query and build query model from it. Secondly, CDB-Server assign those questions generated from model 21 | to different crowdsoucring platforms and collect answers. Finally, it store the query result into table. There are 5 major parts: 22 | 23 | ##### CrowdSQL: 24 | Prase the input query and build syntax tree or graph model. 25 | ##### CrowdExec: 26 | Define the crowd operation of each operator and execute them. 27 | ##### CrowdStorage: 28 | Read data from databases, Store the final result to table, Stash intermediate data 29 | ##### CrowdPlat: 30 | An interface to CDB-Assigment components. 31 | ##### CrowdSchedular: 32 | Schedule operator's execution order 33 | 34 | #### CDB-Assignment 35 | 36 | CDB-Assignment is the assignment module of CrowdDB. 37 | 38 | This module acts as a server to receive requests with well-formatted crowdsourcing tasks data from CDB-Server and publish the tasks to different platforms including AMT (Amazon Mechanical Turk), CF (CrowdFlower), CC (Chinacrowds). 39 | 40 | ## Clone from Github 41 | If you want to get the latest source code, please clone it from Github repo with the following command. 42 | 43 | ``` 44 | git clone https://github.com/TsinghuaDatabaseGroup/CDB.git 45 | cd CDB 46 | ``` 47 | 48 | ## Install 49 | Since CDB has 3 componments, we need install each of them independently. 50 | 51 | ##### Install CDB-Server 52 | Follow this [link](CDB-Server/README.md) 53 | ##### Install CDB-WEB 54 | Follow this [link](CDB-WEB/README.md) 55 | ##### Install CDB-Assignment 56 | Follow this [link](CDB-Assignment/README.md) 57 | 58 | ## Contact 59 | 60 | If you have any questions about it, please feel free to contact Guoliang Li(liguoliang@tsinghua.edu.cn) or Xueping Weng(wxping715@gmail.com). 61 | --------------------------------------------------------------------------------