├── API.md
├── CHANGES
├── Design.md
├── README.md
├── client-doc.md
├── conf
├── jobDescription.md
├── mod.md
└── opencdn.conf
├── database.sql
├── lib
├── OcdnConf.py
├── OcdnJob.py
├── OcdnLogger.py
├── OcdnQueue.py
├── OcdnTask.py
├── cdnconf.py
├── dnserver.py
├── node.py
└── purge.py
└── test
├── consumer.py
└── producer.py
/API.md:
--------------------------------------------------------------------------------
1 | OpenCDN
2 | =======
3 |
4 | Purge
5 | ------
6 | * 模块功能
7 |
8 | 删除指定节点下指定域名的cache
9 |
10 | OpenCDN Purge模块: purge.py
11 |
12 | * Purge API :
13 |
14 | http://node_ip:node_port/ocdn/purge/purge?token=node_token&domain=node_domain
15 |
16 | * 请求参数
17 |
18 | | 变量 | 含义
19 | | ----------- |:-------------:
20 | | node_ip | 节点IP
21 | | node_port | 端口80或者9242
22 | | node_token | 节点token
23 | | node_domain | purge的域名
24 |
25 | * 示例:
26 |
27 | http://192.168.1.1:80/ocdn/purge/purge?token=node_token&domain=ocdn.me
28 |
29 | * 返回参数参考
30 |
31 | Proxy
32 | -----
33 | * 模块功能
34 |
35 | 1. 检测域名反向代理是否生效
36 | 2. 驱动DNS处理模块
37 |
38 | OpenCDN Proxy模块: proxy.py
39 |
40 | * Proxy检测 API :
41 |
42 | http://node_ip/node_path
43 |
44 | 示例:
45 |
46 |
47 |
48 | * 请求参数
49 |
50 | | 变量 | 含义
51 | | ----------- |:-------------:
52 | | node_ip | 节点IP
53 | | node_path | 检测路径
54 |
55 | * 返回参数参考
56 |
57 | DNS
58 | ---
59 | * 模块功能
60 |
61 | 1. 使得DNS失效或者撤销DNS配置
62 |
63 | OpenCDN DNS模块:
64 |
65 | * Proxy检测 API :
66 |
67 |
68 |
69 | 示例:
70 |
71 |
72 |
73 | * 请求参数
74 |
75 | | 变量 | 含义
76 | | ----------- |:-------------:
77 | | node_ip | 节点IP
78 | | node_path | 检测路径
79 |
80 | * 返回参数参考
81 |
82 |
83 | ## 守护模块
84 | Heartbeat
85 | ----
86 | * 模块功能
87 |
88 | 1. 检测所有Node健康状态
89 |
90 | OpenCDN heartbeat模块:
91 |
92 | * Heartbeat检测 API :
93 |
94 | http://node_ip:node_port/ocdn/status?token=node_token
95 |
96 |
97 | * 请求参数
98 |
99 | | 变量 | 含义
100 | | ----------- |:-------------:
101 | | node_ip | 节点IP
102 | | node_port | 端口80或者9242
103 | | node_token | 节点token
104 |
105 | * 示例:
106 |
107 | http://192.168.1.1:80/ocdn/status?token=node_token
108 |
109 | * 返回参数参考
110 |
111 |
112 | FLOW
113 | ----
114 | * 模块功能
115 |
116 | 1. 获取所有Node上域名流量
117 |
118 | OpenCDN flow模块:
119 |
120 | * Flow检测 API :
121 |
122 |
123 | 示例:
124 |
125 |
126 | * 请求参数
127 |
128 | | 变量 | 含义
129 | | ----------- |:-------------:
130 | | node_ip | 节点IP
131 | | node_port | 端口80或者9242
132 | | node_token | 节点token
133 |
134 | * 返回参数参考
135 |
136 |
137 |
--------------------------------------------------------------------------------
/CHANGES:
--------------------------------------------------------------------------------
1 | Changes with OpenCDN-3.1.0 26 Mar 2014
2 |
--------------------------------------------------------------------------------
/Design.md:
--------------------------------------------------------------------------------
1 | OpenCDN Design
2 | --------------
3 |
4 | ### OpenCDN Job和Task
5 |
6 | 一个Job可以是一个Task，也可以是多个Task。
7 |
8 | 1. 增加一个域名
9 | 1. 分发配置文件到CDN节点(ocdn_config)
10 | 2. 节点reload(ocdn_config)
11 | 3. 反向代理检测(ocdn_proxy)
12 | 4. 修改DNS记录(ocdn_dns)
13 | 2. 删除一个域名
14 | 1. 修改DNS记录(ocdn_dns)
15 | 2. 删除配置文件(ocdn_config)
16 | 3. 节点Roload
17 | 4. 擦除Cache(ocdn_purge)
18 | 3. 擦除Cache(ocdn_purge)
19 | 4. 节点流量采集(ocdn_flow)
20 | 5. 节点健康状态检测(ocdn_heartbeat)
21 | 6.
22 |
23 | ### 守护模块
24 | * ocdn_timer 建立一个timer的表，把需要执行的模块和参数放入其中，然后让timer模块来轮询并且触发。
25 | * ocdn_purge 用于刷新缓存<->节点purge
26 | * dns_mode 用于DNS记录变化<->dns服务器或者dnspod等dns服务
27 | * ocdn_config 用户配置文件变更<->节点配置文件
28 | * ocdn_cfg_check(定时) 用户配置文件版本检查<->节点配置文件版本信息(如果有问题则调用ocdn_config进行同步或者调用ocdn_node进行节点排查)注：如果有问题不反复重试，等待定时器的下次检查。
29 | * ocdn_node(定时) 用于检测节点<->节点
30 | * ocdn_flow 流量收集模块(不直接连外网，通过ocdn_node拉取传递过来)
31 |
32 | ### 如何执行一个Job
33 |
34 | 1. OpenCDN中基本执行单元都是Task，一个Job可以包含一个或多个Task。
35 | 2. 每个Job都有一个Json文件来描述所有Task执行流程以及Task之间的数据传输。
36 | 3. 每一种Task都有一个队列，队列的消费者是一启动就存在的。
37 |
38 | 比如执行一个增加域名的Job，那么有几个Task组成一个有向图(DAG)，通过下面来执行。
39 |
40 | > 注：所有Task都已经运行，每个Task都有自己一个队列，所做的就是消费队列中的任务。
41 |
42 |
43 | {
44 | 'TaskName' : 'ADD_DOMAIN', #Task名字
45 | 'Description' : 'add a Domain' # Job所描述
46 | 'TaskList' : ['OCDN_PUSH_CONF','OCDN_RELOAD','OCDN_PROXY','OCDN_DNS'], #Job所有Task列表
47 | 'CurrentTask': 'OCDN_PUSH_CONF', #当前要执行的Task,
48 | 'TimeOut' : 10 #10秒,
49 | 'RunTimesLimit': #Task运行最大次数
50 | {
51 | 'AlreadyRunTimes': 0, #当前执行该Task次数
52 | 'MaxRunTimes' : 10 #最多执行该Task次数
53 | },
54 | 'Parameters': #参数
55 | {
56 | 'Domain':'www.firefoxbug.com',
57 | 'Node':'192.168.1.1'
58 | }
59 | }
60 |
61 | 三个问题大家可以讨论下:
62 |
63 | 1. 如果上面的任意一个Task失败了，怎么确保之前已经成功的Task不重做，并且又能让这次Task重新调度，以及还要保证这个Task不是一个死循环(每次都是失败就有问题了)
64 |
65 | 解决方案：引入了RunTimesLimit这个配置，如果Task执行失败，可以重新调度，AlreadyRunTimes加1，直到超过MaxRunTimes后就判定Task为失败。
66 |
67 | 2. 任务堆积功能考虑，对于cdn节点，挂掉是常态，而这个会导致任务的不一致性。一种是采用队列进行堆积任务，可能数据会太热。有些节点挂几个月，就会占用很多的热存储资源。另一种是等节点重新起来之后再去找哪些任务没做过，然后拉出来做。这个会导致编程的逻辑过于复杂。我考虑的一种方案是，在本地建一个虚拟节点，所有的配置文件数据全部旁路传一份到虚拟节点。然后一旦一个节点挂掉了的话，从虚拟节点rsync(或者类似的原理)一份到实体节点，然后就迅速追回到最新状态开始工作。
68 |
69 |
70 | ### 更多问题可以提出来，可以在上面直接修改，注意增加注释
71 |
72 |
73 |
74 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | OpenCDN
2 | =======
3 |
4 | OpenCDN是一套快速部署CDN加速的工具，针对专门提供CDN加速服务的企业或对多节点CDN加速有需求的企业，提供一套便捷的CDN加速管理平台，可对每一个节点的状态、系统负载进行实时监测与统一管理，同时我们预制了多套常用缓存规则，支持多种复杂的CDN缓存场景。
5 |
6 |
7 | Documentation
8 | =============
9 |
10 | Installation instructions, getting started guides, and in-depth API
11 | documention.
12 |
13 | http://ocdn.me
--------------------------------------------------------------------------------
/client-doc.md:
--------------------------------------------------------------------------------
1 | # OpenCDN Client 标准接口 1.0
2 |
3 | - 所有的信息全部固化在sqlite中
4 | - 操作存在队列的概念，每次只有一个在进行执行
5 | - 所有操作接口皆为全量接口
6 |
7 | ## 标准输出
8 |
9 | {"code": 0, "result": []}
10 |
11 | - version 获取接口版本
12 | - status 当前CDN状态
13 | - webserver/cacheserver/apps/domains/activeLeft/
14 | - notify 通知(如果用户不传入callback,则信息全部放入通知中) 默认最多1000条
15 | - notify/marks 传入id来标记消息已读
16 | - notify/gets 按照id来进行读取
17 | - notify/limit 按照偏移值来进行读取
18 |
19 | - systems 获取所有系统信息
20 | - systems/gets 按照key来获取多项系统信息
21 | - system/cpu 获取系统CPU信息
22 | - system/load 获取系统负载信息
23 | - system/networks 获取系统网卡信息
24 | - system/time 获取系统时间
25 | - configs 获取所有配置信息
26 | - configs/gets 按照key来获取多项配置信息
27 | - configs/updates 配置批量更新 JSON
28 | - configs/limit 用start,limit来获得配置信息
29 | - config/webserver web服务器
30 | - path web服务器路径
31 | - name web服务器软件名称 nginx/tengine/haproxy/ats/lighted/apache
32 | - config/apps 应用
33 | - activeTime 应用生效时间，如果为0，则是立即生效
34 | - config/cacheserver 缓存服务器
35 | - path 缓存服务器路径
36 | - name 缓存服务器软件名称 squid/varnish/nginx_proxy
37 | - apps 获取所有应用名
38 | - apps/gets 获取指定appName的应用
39 | - apps/exists 查找是否存在
40 | - apps/limit 用start,limit来获得应用信息
41 | - apps/deletes 应用批量删除 JSON
42 | - apps/updates 应用批量更新 JSON
43 | - apps/inserts 应用批量添加 JSON
44 | - app/appName 获取应用信息
45 | - app/appName/aliases 获取/更改 应用的别名 ["www.baidu.com", "www.taobao.com"]
46 | - app/appName/sources 获取/更改 应用的源站 [{"address":"1.1.1.1:81", "host": ""}]
47 | - app/appName/caches 获取/更改 应用的缓存策略 [{"type": "suffix", ""},]
48 | - actions
49 | - actions.start CDN服务启动
50 | - actions.stop CDN服务停止
51 | - actions.restart CDN服务重启
52 | - actions.refresh 全局刷新配置文件并自动reload
53 |
54 |
55 | ## web
56 |
57 | 管理界面
58 |
--------------------------------------------------------------------------------
/conf/jobDescription.md:
--------------------------------------------------------------------------------
1 |
2 | | TaskName | 含义
3 | | --------------------|:-------------:
4 | | OCDN_ADD_NODE | 添加节点
5 | | OCDN_DEL_NODE | 添加节点
6 | | OCDN_ADD_DOMAIN | 添加域名
7 | | OCDN_DEL_DOMAIN | 删除域名
8 | | OCDN_REMOVE_CONF | 删除配置文件
9 | | OCDN_SYNC_CONF | 同步配置文件
10 | | OCDN_VER_CONF | 查看配置文件的版本
11 | | OCDN_RELOAD_NODE | 节点reload
12 | | OCDN_PROXY | 节点proxy检测
13 | | OCDN_ADD_DNS | 增加DNS
14 | | OCDN_DEL_DNS | 撤销DNS
15 | | OCDN_PURGE | 擦除缓存
16 | | OCDN_NODE_MONITOR | 节点检查检查、配置一致性检查
17 | | OCDN_TASK_DISPATCH | 根据节点一致性问题分发相应Job
18 | | OCDN_LOOP_TASK | 无尽循环任务,会定时会将TaskList自动提交
19 |
20 | ### 增加域名
21 |
22 | {
23 | 'JobName' : 'OCDN_ADD_DOMAIN', #Job名字
24 | 'Description' : 'add a Domain', # Job所描述
25 | 'TaskList' : ['OCDN_PUSH_CONF','OCDN_RELOAD_NODE','OCDN_PROXY','OCDN_ADD_DNS'], #Job所有Task列表
26 | 'CurrentTask': 'OCDN_PUSH_CONF', #当前要执行的Task,
27 | 'TimeOut' : 10 #10秒,
28 | 'RunTimesLimit': #Task运行最大次数
29 | {
30 | 'AlreadyRunTimes': 0, #当前执行该Task次数
31 | 'MaxRunTimes' : 10 #最多执行该Task次数
32 | },
33 | 'Parameters': #参数
34 | {
35 | 'Domain':'www.firefoxbug.com',
36 | 'Node':['192.168.1.1','192.168.1.2']
37 | }
38 | }
39 |
40 |
41 | ### 删除域名
42 |
43 | {
44 | 'JobName' : 'OCDN_DEL_DOMAIN', #Job名字
45 | 'Description' : 'del a Domain', # Job所描述
46 | 'TaskList' : ['OCDN_DEL_DNS'，'OCDN_DEL_CONF','OCDN_RELOAD'], #Job所有Task列表
47 | 'CurrentTask': 'OCDN_PUSH_CONF', #当前要执行的Task,
48 | 'TimeOut' : 10 #10秒,
49 | 'RunTimesLimit': #Task运行最大次数
50 | {
51 | 'AlreadyRunTimes': 0, #当前执行该Task次数
52 | 'MaxRunTimes' : 10 #最多执行该Task次数
53 | },
54 | 'Parameters': #参数
55 | {
56 | 'Domain':'www.firefoxbug.com',
57 | 'Node':['192.168.1.1','192.168.1.2']
58 | }
59 | }
60 |
61 |
62 | ### 增加节点
63 |
64 | {
65 | 'JobName' : 'OCDN_ADD_NODE', #Job名字
66 | 'Description' : 'del a Node', # Job所描述
67 | 'TaskList' : ['OCDN_ADD_NODE'，'OCDN_SYNC_CONF','OCDN_RELOAD_NODE'], #Job所有Task列表
68 | 'CurrentTask': 'OCDN_ADD_NODE', #当前要执行的Task,
69 | 'TimeOut' : 10 #10秒,
70 | 'RunTimesLimit': #Task运行最大次数
71 | {
72 | 'AlreadyRunTimes': 0, #当前执行该Task次数
73 | 'MaxRunTimes' : 10 #最多执行该Task次数
74 | },
75 | 'Parameters': #参数
76 | {
77 | 'Node':'192.168.1.1'
78 | }
79 | }
80 |
81 |
82 | ### 撤销DNS
83 |
84 | {
85 | 'JobName' : 'OCDN_DEL_DNS', #Job名字，由于节点故障而撤回DNS
86 | 'Description' : 'del a Node', # Job所描述
87 | 'TaskList' : ['OCDN_DEL_DNS'], #Job所有Task列表
88 | 'CurrentTask': 'OCDN_DEL_DNS', #当前要执行的Task,
89 | 'TimeOut' : 10 #10秒,
90 | 'RunTimesLimit': #Task运行最大次数
91 | {
92 | 'AlreadyRunTimes': 0, #当前执行该Task次数
93 | 'MaxRunTimes' : 10 #最多执行该Task次数
94 | },
95 | 'Parameters': #参数
96 | {
97 | ['Node':'192.168.1.1']
98 | }
99 | }
100 |
101 |
102 | ### Purge cache
103 |
104 | {
105 | 'JobName' : 'OCDN_PURGE', #Job名字
106 | 'Description' : 'purge a Domain cache',
107 | 'TaskList' : ['OCDN_PURGE'],
108 | 'CurrentTask': 'OCDN_PURGE', #当前要执行的Task,
109 | 'TimeOut' : 10 #10秒,
110 | 'RunTimesLimit': #Task运行最大次数
111 | {
112 | 'AlreadyRunTimes': 0, #当前执行该Task次数
113 | 'MaxRunTimes' : 10 #最多执行该Task次数
114 | },
115 | 'Parameters': #参数
116 | {
117 | [
118 | {
119 | 'ip':'192.168.1.1',port':'80','domain':'www.firefoxbug.com',token':'821e57c57e8455e3e809e23df7bb6ce9'
120 | },
121 | {
122 | 'ip':'192.168.1.2',port':'80','domain':'www.firefoxbug.com',token':'821e57c57e8455e3e809e23df7bb6ce9'
123 | }
124 | ]
125 | }
126 | }
127 |
128 |
129 | ###节点健康检查
130 |
131 | {
132 | 'JobName' : 'OCDN_NODE_CHECKER', #Job名字
133 | 'Description' : 'Check cdn node healthy and recover Bad node's configurations',
134 | 'TaskList' : ['OCDN_NODE_MONITOR','OCDN_TASK_DISPATCH','OCDN_LOOP_TASK'],
135 | 'CurrentTask': 'OCDN_NODE_MONITOR', #当前要执行的Task,
136 | 'TimeOut' : 600 #10秒,
137 | 'RunTimesLimit': #Task运行最大次数
138 | {
139 | 'AlreadyRunTimes': 0, #当前执行该Task次数
140 | 'MaxRunTimes' : 10 #最多执行该Task次数
141 | },
142 | 'Parameters': #参数
143 | {
144 | #OLD_OK表示上一次检查的结果,经过OCDN_NODE_MONITOR后，OK->BAD的变成 NEW_BAD;BAD-> OK 变成NEW_OK
145 | #OCDN_NODE_MONITOR 执行完后 将节点状态的json一次性持久华到DB,以供状态查询
146 | 'NODE_STATUS':[
147 | 'OK': ['192.168.1.1'],
148 | 'BAD':['192.168.1.2']
149 | 'NEW_OK':[]
150 | 'NEW_BAD':[]
151 | ],
152 | #OCDN_TASK_DISPATCH 会找NEW_OK或NEW_BAD的机器，提交相应的Job到相应的Queue
153 | 'LOOP_SLEEP_TIME':600
154 | #OCDN_LOOP_TASK 则是Sleep LOOP_SLEEP_TIME 秒后，将本json重新提交到本Queue中
155 | }
156 | }
157 |
158 |
--------------------------------------------------------------------------------
/conf/mod.md:
--------------------------------------------------------------------------------
1 | 各个模块参数和功能
2 | ================
3 |
4 | # 模块
5 |
6 | * OCDN_MODIFI_DNS 生效DNS记录
7 |
8 | 函数依赖: 通信
9 |
10 | 传入参数: subDomain, ip
11 |
12 | 业务逻辑:
13 |
14 |
15 | * OCDN_DEL_DNS
16 |
17 | 函数依赖: 无
18 |
19 | 传入参数: subDomain, ip or record_id
20 |
21 | 业务逻辑:
22 |
23 |
24 | * OCDN_SYNC_CONF 同步配置文件
25 |
26 | 函数依赖: 通信
27 |
28 | 传入参数: config, file, ip, token
29 |
30 |
31 | * OCDN_DEL_CONF 删除配置文件
32 |
33 | 函数依赖: 通信
34 |
35 | 传入参数: file, ip, token
36 |
37 |
38 | * OCDN_MAKE_CONF 生成配置文件
39 |
40 | 函数依赖: db
41 |
42 | 传入参数: domain
43 |
44 |
45 | * OCDN_VER_CONF 检查配置文件版本
46 |
47 | 函数依赖: 通信
48 |
49 | 传入参数: domain, ip, version
50 |
51 |
52 | * OCDN_TRAFFIC 通信模块
53 |
54 | 函数依赖: 无
55 |
56 | 传入参数: http包(method,url,data...)
57 |
58 |
59 | * OCDN_DOMAIN_STATUS 域名状态更新
60 |
61 | 函数依赖: db
62 |
63 | 传入参数: domain, status(normal,complete,deleting,delete)
64 |
65 | 业务逻辑: 控制数据库domain记录变更状态
66 |
67 | 返回值: nodes
68 |
69 |
70 | * OCDN_NODE_RELOAD 节点reload
71 |
72 | 函数依赖: 通信
73 |
74 | 传入参数: ip, token
75 |
76 |
77 | * OCDN_DOMAIN_NODE 域名节点状态更新
78 |
79 | 函数依赖: db
80 |
81 | 传入参数: domain, ip, status(new,complete,fault,delete)
82 |
83 |
84 | * OCDN_NODE_STATUS 节点状态更新
85 |
86 | 函数依赖: db
87 |
88 | 传入参数: ip, status(normal, fault, deleting, delete)
89 |
90 | 业务逻辑: 控制数据node记录变更状态
91 |
92 | 返回值: domains
93 |
94 |
95 | * OCDN_NODE_HEART 节点检测
96 |
97 | 函数依赖: 通信
98 |
99 | 传入参数: ip, token
100 |
101 |
102 | # 任务
103 |
104 | * ADD_DOMAIN
105 |
106 | ** OCDN_DOMAIN_STATUS(domain, status=normal) -> get nodes
107 |
108 | ** OCDN_MAKE_CONF(domain)
109 |
110 | ** OCDN_SYNC_CONF(config, file, ip, token)
111 |
112 | ** OCDN_NODE_RELOAD(ip, token) -> (No) retry..
113 |
114 | ** OCDN_VER_CONF(domain, ip, version) -> (No) retry..
115 |
116 | ** OCDN_MODIFI_DNS(subDomain, ip) -> (No) retry..
117 |
118 | ** OCDN_DOMAIN_STATUS(domain, status=complete)
119 |
120 | * REMOVE_DOMAIN
121 |
122 | ** OCDN_DOMAIN_STATUS(domain, status=deleting) -> get nodes
123 |
124 | ** OCDN_DEL_DNS(subDomain, ip) -> (No) retry..
125 |
126 | ** OCDN_DEL_CONF(file, ip, token) -> (No) retry..
127 |
128 | ** OCDN_DOMAIN_NODE(domain, ip, status=delete)
129 |
130 | ** OCDN_DOMAIN_STATUS(domain, status=delete)
131 |
132 |
133 | * ADD_NODE
134 |
135 | ** OCDN_NODE_HEART(ip, token)
136 |
137 | ** OCDN_NODE_STATUS(ip, status=normal)
138 |
139 |
140 | * REMOVE_NODE
141 |
142 | ** OCDN_NODE_STATUS(ip, status=deleting) -> get domains
143 |
144 | ** OCDN_DEL_DNS(suDomain, ip) -> (No) retry..
145 |
146 | ** OCDN_DEL_CONF(file, ip, token) -> (No) retry..
147 |
148 | ** OCDN_DOMAIN_NODE(domain, ip, status=delete)
149 |
150 | ** OCDN_NODE_STATUS(ip, status=delete)
151 |
152 |
153 | * FAULT_NODE 当一个节点故障
154 |
155 | ** OCDN_NODE_STATUS(ip, status=fault) -> get domains
156 |
157 | ** OCDN_DEL_DNS(suDomain, ip) -> (No) retry..
158 |
159 |
160 | * RECOVER_NODE 当一个节点恢复
161 |
162 | ** OCDN_NODE_STATUS(ip) -> get domains
163 |
164 | ** OCDN_SYNC_CONF(config, file, ip, token)
165 |
166 | ** OCDN_MODIFI_DNS(suDomain, ip) -> (No) retry..
167 |
168 | ** OCDN_NODE_STATUS(ip, status=normal)
169 |
170 |
171 | * JOIN_NODE 把一个节点加入一个已经在运行的CDN域名
172 |
173 | ** OCDN_DOMAIN_NODE(domain, ip, status=new)
174 |
175 | ** OCDN_SYNC_CONF(config, file, ip, token)
176 |
177 | ** OCDN_MODIFI_DNS(suDomain, ip) -> (No) retry..
178 |
179 | ** OCDN_DOMAIN_NODE(domain, ip, status=complete)
180 |
181 |
182 |
183 |
--------------------------------------------------------------------------------
/conf/opencdn.conf:
--------------------------------------------------------------------------------
1 | [module]
2 |
3 |
4 | [mysql]
5 |
6 |
7 | [gearman]
8 | queue_ip = "127.0.0.1"
9 | queue_port = 4730
10 |
11 | [redis]
12 | redis_ip = "127.0.0.1"
13 | redis_port = 6379
--------------------------------------------------------------------------------
/database.sql:
--------------------------------------------------------------------------------
1 | CREATE TABLE IF NOT EXISTS `domains` (
2 | `domain_id` int(10) unsigned NOT NULL AUTO_INCREMENT,
3 | `domain` varchar(255) NOT NULL,
4 | `ext` tinyint(1) NOT NULL DEFAULT '0' COMMENT '0:普通域名,1:泛域名',
5 | `create_time` int(10) unsigned NOT NULL DEFAULT '0',
6 | `last_update_time` int(10) unsigned NOT NULL DEFAULT '0',
7 | PRIMARY KEY (`domain_id`),
8 | UNIQUE KEY `name` (`domain`)
9 | ) ENGINE=MyISAM DEFAULT CHARSET=utf8;
10 |
11 | CREATE TABLE IF NOT EXISTS `routers` (
12 | `router_id` int(10) unsigned NOT NULL AUTO_INCREMENT,
13 | `domain_id` int(10) unsigned NOT NULL,
14 | `source_id` int(10) unsigned NOT NULL,
15 | `server_id` int(10) unsigned NOT NULL,
16 | `source_mode` tinyint(1) unsigned NOT NULL COMMENT '0:master,1:backup,2:hash',
17 | `create_time` int(10) unsigned NOT NULL DEFAULT '0',
18 | `last_update_time` int(10) unsigned NOT NULL DEFAULT '0',
19 | PRIMARY KEY (`router_id`),
20 | UNIQUE KEY `domain_id` (`domain_id`,`source_id`,`server_id`)
21 | ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
22 |
23 | CREATE TABLE IF NOT EXISTS `servers` (
24 | `server_id` int(10) unsigned NOT NULL AUTO_INCREMENT,
25 | `ip` varchar(255) NOT NULL,
26 | `token` char(32) NOT NULL,
27 | `last_update_time` int(10) unsigned NOT NULL DEFAULT '0',
28 | `create_time` int(10) unsigned NOT NULL DEFAULT '0',
29 | PRIMARY KEY (`server_id`)
30 | ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
31 |
32 | CREATE TABLE IF NOT EXISTS `sources` (
33 | `source_id` int(10) unsigned NOT NULL AUTO_INCREMENT,
34 | `domain_id` int(10) unsigned NOT NULL,
35 | `address` varchar(255) NOT NULL COMMENT '源站IP或者域名',
36 | `create_time` int(10) unsigned NOT NULL DEFAULT '0',
37 | `last_update_time` int(10) unsigned NOT NULL DEFAULT '0',
38 | PRIMARY KEY (`source_id`),
39 | KEY `domain_id` (`domain_id`)
40 | ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
41 |
--------------------------------------------------------------------------------
/lib/OcdnConf.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python
2 | # -*- coding: UTF-8 -*-
3 |
4 | # author : firefoxbug
5 | # E-Mail : wanghuafire@gmail.com
6 | # Blog : www.firefoxbug.com
7 |
8 | """ OpenCDN Conf module: Parse opencdn configure file.
9 |
10 | Conf file:
11 | conf/opencdn.conf
12 | """
13 |
14 | import ConfigParser
15 | import string, os, sys
16 |
17 | parent, bindir = os.path.split(os.path.dirname(os.path.abspath(sys.argv[0])))
18 | if os.path.exists(os.path.join(parent, 'conf')):
19 | sys.path.insert(0, os.path.join(parent, 'conf'))
20 |
21 | class MyParseConf(object):
22 | def __init__(self,conf_path):
23 | self.conf_path = conf_path
24 | self.cf = ConfigParser.ConfigParser()
25 | self.cf.read(conf_path)
26 |
27 | def get_value(self,section,key,is_bool = False,is_int = False):
28 | if is_bool and not is_int:
29 | #bool类型
30 | value = self.cf.getboolean(section,key)
31 | print key,":",value,type(value)
32 | return value
33 | elif not is_bool and is_int:
34 | value = self.cf.getint(section,key)
35 | print key,":",value,type(value)
36 | return value
37 |
38 | value = self.cf.get(section,key)
39 | print key,":",value,type(value)
40 | return value
41 |
42 | class OcdnParseConf(object):
43 | """docstring for OcdnParseConf"""
44 | def __init__(self):
45 | self.conf_path = os.path.join(parent, 'conf/opencdn.conf')
46 | if not os.path.exists(self.conf_path) :
47 | print "ERROR: opencdn.conf does not exists."
48 | sys.exit(255)
49 | self.mpc = MyParseConf(self.conf_path)
50 |
51 | def get_queue_module(self):
52 | """Parse opencdn.conf and get queue ip and queue port"""
53 | queue_ip = self.mpc.get_value("gearman", "queue_ip")
54 | queue_port = self.mpc.get_value("gearman", "queue_port", is_int=True)
55 | print queue_ip, queue_port
56 |
57 |
58 | if __name__ == '__main__':
59 | parser = OcdnParseConf()
60 | parser.get_queue_module()
61 |
--------------------------------------------------------------------------------
/lib/OcdnJob.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python
2 | # -*- coding: UTF-8 -*-
3 |
4 | # author : firefoxbug
5 | # E-Mail : wanghuafire@gmail.com
6 | # Blog : www.firefoxbug.com
7 |
8 | """Run a job's tasklist in order.
9 |
10 | JobManagero: contorl job run
11 | """
12 | import time
13 | import random
14 |
15 | class JobManager(object):
16 | """Manager a job's tasks run orger
17 |
18 | """
19 | def __init__(self, jobjson):
20 | self.jobjson = jobjson
21 | self.task_lists = None
22 | self.CurrentTask = None
23 | self.Parameters = None
24 | self.AlreadyRunTimes = None
25 | self.MaxRunTimes = None
26 |
27 | def check_job_json_is_ok(self):
28 | """Check the job json file syntax is ok or not
29 |
30 | return True if the json is ok
31 | return False if the json is error
32 | """
33 | try:
34 | self.CurrentTask = self.jobjson['CurrentTask']
35 | self.task_lists = self.jobjson['TaskList']
36 | self.Parameters = self.jobjson['Parameters']
37 | self.AlreadyRunTimes = int(self.jobjson['RunTimesLimit']['AlreadyRunTimes'])
38 | self.MaxRunTimes = int(self.jobjson['RunTimesLimit']['MaxRunTimes'])
39 | if not self.CurrentTask or not self.task_lists :
40 | return False
41 | return True
42 | except Exception, e:
43 | print str(e)
44 | return False
45 |
46 | def get_task_id(self):
47 | """Generate a Task ID
48 |
49 | return (YearMonthDayHourMinuteSecond)
50 | """
51 | return time.strftime('%Y%m%d%H%M%S',time.localtime(time.time()))+str(random.randint(0,10))
52 |
53 | def get_current_task_to_run(self):
54 | """Get current task is goting to run
55 |
56 | Return (TaskName, Parameters)
57 | """
58 | return (self.CurrentTask, self.Parameters)
59 |
60 | def get_current_task_parameters(self):
61 | """Get current task's parameters
62 |
63 | Return (**args)
64 | """
65 | pass
66 |
67 | def is_job_finished(self):
68 | """If current task is the last one in this job
69 |
70 | Return True: means the job is finished or some errors
71 | Return False: means the job is unfinished
72 | """
73 | if self.CurrentTask == self.task_lists[-1]:
74 | return True
75 | else :
76 | return False
77 |
78 | def set_next_task_to_run(self, parameters=[]):
79 | """Valid next task going to run's json
80 |
81 | Return json
82 | """
83 | self.jobjson['CurrentTask'] = self.get_next_task_module_name()
84 | if not self.jobjson['CurrentTask']:
85 | return None
86 | self.jobjson['RunTimesLimit']['AlreadyRunTimes'] = 0
87 | self.jobjson['Parameters'] = parameters
88 | return self.jobjson
89 |
90 | def get_next_task_parameters(self):
91 | """Set Next task's parameters
92 |
93 | """
94 | pass
95 |
96 | def get_next_task_module_name(self):
97 | """Get next to run's task module name
98 |
99 | Return string type
100 | eg. OCDN_PUSH_CONF
101 | """
102 | try:
103 | current_task_index = self.task_lists.index(self.CurrentTask)
104 | return self.task_lists[current_task_index+1]
105 | except Exception, e:
106 | return None
107 |
108 | def set_run_time_out(self, timeout=30):
109 | """Set task run out time, default is 30 s.
110 |
111 | """
112 |
113 | def try_run_current_task_again(self):
114 | """run this current task again
115 |
116 | return None: the task run times is exceed
117 | return Json: the task is going to dispatched again
118 | """
119 | if self.is_task_run_times_exceed():
120 | return None
121 |
122 | self.increase_task_run_times()
123 | return self.jobjson
124 |
125 | def is_task_run_times_exceed(self):
126 | """If this tasks has been dispatched more then the limits, mark this job failured and
127 | do not dispatch again.
128 |
129 | Return True : mark the job failured
130 | Return False : the job will be dispatched again
131 | """
132 | if self.AlreadyRunTimes >= self.MaxRunTimes:
133 | return True
134 | return False
135 |
136 | def increase_task_run_times(self):
137 | """Because this task runed failured, set number 'AlreadyRunTimes' plus 1.
138 | """
139 | self.jobjson['RunTimesLimit']['AlreadyRunTimes'] = self.AlreadyRunTimes + 1
140 |
141 | class OcdnJSON(object):
142 | """Create job json description"""
143 | def __init__(self):
144 | self.json_template = {
145 | 'JobName' : None,
146 | 'Description' : 'OpenCDN',
147 | 'TaskList' : [],
148 | 'CurrentTask': None,
149 | 'TimeOut' : 10,
150 | 'RunTimesLimit':
151 | {
152 | 'AlreadyRunTimes': 0,
153 | 'MaxRunTimes' : 10
154 | },
155 | 'Parameters':{
156 |
157 | }
158 | }
159 |
160 | def create_job_json(self, JobName, TaskList=[], Parameters=[]):
161 | self.json_template['JobName'] = JobName
162 | self.json_template['TaskList'] = TaskList
163 | self.json_template['Parameters'] = Parameters
164 | try:
165 | self.json_template['CurrentTask'] = TaskList[0]
166 | except Exception, e:
167 | return None
168 |
169 | return self.json_template
170 |
--------------------------------------------------------------------------------
/lib/OcdnLogger.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/python
2 | # -*- coding: UTF-8 -*-
3 |
4 | import logging
5 |
6 | """
7 | log level:
8 | NOTSET < DEBUG < INFO < WARNING < ERROR < CRITICAL
9 |
10 | logging.DEBUG
11 | logging.INFO
12 | logging.WARNING
13 | logging.ERROR
14 | logging.CRITICAL
15 | """
16 |
17 | def init_logger(logfile='test.log', logmodule='OpenCDN', stdout=True):
18 | formatter = logging.Formatter('%(levelname)s [%(asctime)s] [%(name)s]:%(pathname)s line=%(lineno)d [message="%(message)s"]')
19 |
20 | logger = logging.getLogger(logmodule)
21 | logger.setLevel(logging.INFO)
22 |
23 | # write log into file
24 | fh = logging.FileHandler(logfile)
25 | fh.setLevel(logging.INFO)
26 | fh.setFormatter(formatter)
27 | logger.addHandler(fh)
28 |
29 | if stdout:
30 | # print log to stdout
31 | ch = logging.StreamHandler()
32 | ch.setLevel(logging.INFO)
33 | ch.setFormatter(formatter)
34 | logger.addHandler(ch)
35 | return logger
36 |
37 | if __name__ == '__main__':
38 | logger = init_logger(logfile='debug.log', stdout=True)
39 | logger.info('this is a info log')
40 | try:
41 | print 1/0
42 | except Exception, e:
43 | logger.error(str(e[0]))
44 |
--------------------------------------------------------------------------------
/lib/OcdnQueue.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python
2 | # -*- coding: UTF-8 -*-
3 |
4 | # author : firefoxbug
5 | # E-Mail : wanghuafire@gmail.com
6 | # Blog : www.firefoxbug.com
7 |
8 | """ OpenCDN task queue module: manage all tasks queue.
9 |
10 | Queue Task Name:
11 |
12 | | OCDN_ADD_NODE | 添加节点
13 | | OCDN_DEL_NODE | 添加节点
14 | | OCDN_ADD_DOMAIN | 添加域名
15 | | OCDN_DEL_DOMAIN | 删除域名
16 | | OCDN_REMOVE_CONF | 删除配置文件
17 | | OCDN_SYNC_CONF | 同步配置文件
18 | | OCDN_VER_CONF | 查看配置文件的版本
19 | | OCDN_RELOAD_NODE | 节点reload
20 | | OCDN_PROXY | 节点proxy检测
21 | | OCDN_ADD_DNS | 增加DNS
22 | | OCDN_DEL_DNS | 撤销DNS
23 | | OCDN_PURGE | 擦除缓存
24 |
25 | """
26 | import sys
27 | import gearman
28 | try:
29 | import json
30 | except Exception, e:
31 | import simplejson as json
32 | from OcdnLogger import init_logger
33 |
34 | class Producer(object):
35 | """OpenCDN Queue control: use gearman manage all tasks queue.
36 |
37 | producer:
38 | 1. producer need to connect to gearman
39 | 2. produce tasks and put them into queue"""
40 | def __init__(self, queue_ip='127.0.0.1', queue_port=4730):
41 | self.gearman_server_addr = "%s:%s"%(queue_ip,queue_port)
42 | self.producer =None
43 |
44 | def connect_queue(self):
45 | """Producer try to connect to gearman server"""
46 | self.producer = gearman.GearmanClient([self.gearman_server_addr])
47 |
48 | def put_task_into_queue(self, queue_name, data, Background=True):
49 | """Put a task into a queue"""
50 | self.producer.submit_job(queue_name, json.dumps(data), background=Background)
51 |
52 | class Consumer(gearman.GearmanWorker):
53 | """OpenCDN Queue control: use gearman manage all tasks queue.
54 |
55 | consumer:
56 | 1. consumer need to register into gearman
57 | 2. get tasks from queue and dispatch them
58 | """
59 | def __init__(self, queue_ip='127.0.0.1', queue_port=4730, logfile='consumer.log'):
60 | self.gearman_server_addr = "%s:%s"%(queue_ip,queue_port)
61 | super(Consumer, self).__init__([self.gearman_server_addr])
62 | self.conlog = init_logger(logfile=logfile, logmodule='CONSUMER', stdout=True)
63 |
64 | def register_task_callback(self, queue_name, callback):
65 | """Register callback module. once task arrive in the queue the callback
66 | will be called and dispatch the task
67 | """
68 | msg = 'TaskModule:%s'%(queue_name)
69 | try:
70 | self.register_task(queue_name, callback)
71 | self.conlog.info('%s: Register task into gearman success'%(msg))
72 | except Exception, e:
73 | self.conlog.error('%s: Register task into gearman failed: %s'%(msg))
74 |
75 | def start_worker(self):
76 | """Do tasks in a loop"""
77 | self.work()
78 |
79 | def on_job_execute(self, current_job):
80 | return super(Consumer, self).on_job_execute(current_job)
81 |
82 | def on_job_exception(self, current_job, exc_info):
83 | error_msg = 'TaskModule: %s '%current_job.task
84 | for item in exc_info[1]:
85 | error_msg += str(item)
86 | self.conlog.error(error_msg)
87 | return super(Consumer, self).on_job_exception(current_job, exc_info)
88 |
89 | def on_job_complete(self, current_job, job_result):
90 | # message = '[SUCCESS] TaskModule: %s '%(current_job.task)
91 | # self.conlog.info(message)
92 | return super(Consumer, self).send_job_complete(current_job, job_result)
93 |
94 | # @classmethod
95 | def push_task(self, queue_ip, queue_port, queue_name, data, Background=True):
96 | """Connect to queue and put task into queue"""
97 | try:
98 | self.gearman_server_addr = "%s:%s"%(queue_ip,queue_port)
99 | self.producer = gearman.GearmanClient([self.gearman_server_addr])
100 | self.producer.submit_job(queue_name, json.dumps(data), background=Background)
101 | self.conlog.info('TaskModule: %s: put task into queue success'%(queue_name))
102 | return True
103 | except Exception, e:
104 | # print str(e)
105 | self.conlog.error('TaskModule: %s: Put task into queue. Data:%s'%(queue_name, data))
106 | return False
107 |
108 | import redis
109 |
110 | class RedisQueue(object):
111 | """Simple Queue with Redis Backend
112 |
113 | >>> redis_q = RedisQueue(host='127.0.0.1')
114 | >>> redis_q.put('KEY', 'Value')
115 | >>> redis_q.get('KEY')
116 | """
117 | def __init__(self,**redis_kwargs):
118 | """The default connection parameters are: host='localhost', port=6379, db=0"""
119 | self.__db= redis.Redis(**redis_kwargs)
120 |
121 | def qsize(self,key):
122 | """Return the approximate size of the queue."""
123 | return self.__db.llen(key)
124 |
125 | def empty(self):
126 | """Return True if the queue is empty, False otherwise."""
127 | return self.qsize() == 0
128 |
129 | def put(self,key,item):
130 | """Put item into the queue."""
131 | self.__db.rpush(key, item)
132 |
133 | def get(self,key,block=True, timeout=None):
134 | """Remove and return an item from the queue.
135 |
136 | If optional args block is true and timeout is None (the default), block
137 | if necessary until an item is available."""
138 | if block:
139 | item = self.__db.blpop(key, timeout=timeout)
140 | else:
141 | item = self.__db.lpop(key)
142 |
143 | if item:
144 | item = item[1]
145 | return item
146 |
147 | def get_nowait(self):
148 | """Equivalent to get(False)."""
149 | return self.get(False)
150 |
151 | if __name__ == '__main__':
152 | Consumer.push_task(queue_ip='103.6.222.21', queue_port=4730, queue_name='OCDNQUEUE', data='hello')
153 |
--------------------------------------------------------------------------------
/lib/OcdnTask.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python
2 | # -*- coding: UTF-8 -*-
3 |
4 | # author : firefoxbug
5 | # E-Mail : wanghuafire@gmail.com
6 | # Blog : www.firefoxbug.com
7 |
8 | """ OpenCDN Task Moudle: Send url request and get response parallel.
9 |
10 | Dependenciys: gevent
11 | Notice: unknow gevent monkey how works
12 |
13 | """
14 | import gevent.monkey
15 | gevent.monkey.patch_all()
16 |
17 | import gevent
18 | import urllib2
19 |
20 | class RunTask(object):
21 | """Run instance parallel"""
22 | def __init__(self):
23 | super(RunTask, self).__init__()
24 | pass
25 |
26 | def fetchurl(self, task_num, url, timeout=5):
27 | """Send http request and get response no-blocking
28 |
29 | coroutine dispatched may cause all tasks run time increase, so set
30 | 'timeout' larger then you expected.
31 | """
32 | # print('Process %s: %s start work' % (task_num, url))
33 | flag = None
34 | status = None
35 | message = None
36 | try :
37 | req_obj = urllib2.Request(url)
38 | response = urllib2.urlopen(req_obj, timeout=timeout)
39 | status = response.code
40 | # message = response.read()
41 | flag = True
42 | except Exception, e :
43 | message = str(e)
44 | flag = False
45 |
46 | # print('Process %s: %s %s' % (task_num, url, status))
47 | return (task_num, flag, url, status, message)
48 |
49 | def run(self, url_list):
50 | """Gevent spawn multiply coroutines to run parallel
51 |
52 | Return tuple contain list : [(ip, status, reason)..]
53 | eg: [(192.168.1.1, True, None),(192.168.2.2, Flase, 'Timeout')]
54 | """
55 | i = 0
56 | jobs = []
57 | for url in url_list :
58 | jobs.append(gevent.spawn(self.fetchurl, i, url))
59 | i += 1
60 | gevent.joinall(jobs)
61 | # for job in jobs :
62 | # print job.value
63 | return [ job.value for job in jobs]
64 |
65 | if __name__ == '__main__':
66 | """Test gevent"""
67 | url_list = ['http://107.167.184.223/' for _ in range(0, 200)]
68 | tasker = RunTask()
69 | result = tasker.run(url_list)
70 | res = {'SUCCESS':0, 'FAILED':0}
71 | for item in result :
72 | if item[1] :
73 | res['SUCCESS'] += 1
74 | else :
75 | res['FAILED'] += 1
76 | print 'SUCCESS %s '%(res['SUCCESS'])
77 | print 'FAILED %s'%(res['FAILED'])
78 |
--------------------------------------------------------------------------------
/lib/cdnconf.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python
2 | # -*- coding: UTF-8 -*-
3 |
4 | # author : firefoxbug
5 | # E-Mail : wanghuafire@gmail.com
6 | # Blog : www.firefoxbug.com
7 |
8 | """ OpenCDN Conf module: push or remove CDN config files.
9 |
10 | OpenCDN module instruction :
11 | OCDN_PUSH_CONF
12 | OCDN_REMOVE_CONF
13 |
14 | Purge API :
15 |
16 | """
17 |
18 | class Conf(object):
19 | """OpenCDN CDN Nodes config file manager
20 |
21 | 1. push all config files to CDN Nodes.
22 | 2. remove all config files on CDN Nodes.
23 | """
24 | def __init__(self, arg):
25 | super(OcdnConf, self).__init__()
26 | self.arg = arg
27 |
28 | def conf_loop():
29 | while True:
30 | current_task_json_str = Queue.get_item_from_queue('OCDN_PUSH_CONF')
31 | jobjson = JobJSON(current_task_json)
32 | (call_module, **arg) = jobjson.get_current_task_to_run()
33 | if self.push_cdn_conf(**arg) == False:
34 | self.push_cdn_conf_failure()
35 | return False
36 |
37 | jobjson.set_next_task_to_run(parameters)
38 | Queue.put_item_into_queue('OCDN_RELOAD')
39 |
40 | def push_cdn_conf():
41 | """sync nginx config file to all Nodes
42 |
43 | """
44 |
45 | def push_cdn_conf_failure():
46 | """sync config files failed
47 | """
48 |
49 | def remove_cdn_conf():
50 | """remove nginx config file
51 |
52 | """
53 |
54 | def remove_cdn_conf_failure():
55 | """remove nginx config file failed
56 |
57 | """
--------------------------------------------------------------------------------
/lib/dnserver.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python
2 | # -*- coding: UTF-8 -*-
3 |
4 | # author : firefoxbug
5 | # E-Mail : wanghuafire@gmail.com
6 | # Blog : www.firefoxbug.com
7 |
8 | """ OpenCDN DNS module: add or remove dns record.
9 |
10 | OpenCDN module instruction :
11 | OCDN_ADD_DNS
12 | OCDN_DEL_DNS
13 |
14 | Reload API :
15 | """
16 |
17 | class DNS(object):
18 | """OpenCDN DNS add or remove"""
19 | def __init__(self, arg):
20 | super(DNS, self).__init__()
21 | self.arg = arg
22 |
23 | def append_record():
24 | """Add A record
25 | """
26 | pass
27 |
28 | def append_record_failure():
29 | """Add A record failued
30 | """
31 | pass
32 |
33 | def remove_record():
34 | """Remove A record
35 | """
36 | pass
37 |
38 | def remove_record_failure():
39 | """Remove A record failued
40 | """
41 | pass
--------------------------------------------------------------------------------
/lib/node.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python
2 | # -*- coding: UTF-8 -*-
3 |
4 | # author : firefoxbug
5 | # E-Mail : wanghuafire@gmail.com
6 | # Blog : www.firefoxbug.com
7 |
8 | """ OpenCDN Node module: OpenCDN Node manager, reload CDN nodes.
9 |
10 | OpenCDN module instruction :
11 | OCDN_RELOAD
12 |
13 | Reload API :
14 |
15 | """
16 |
17 | class Node(object):
18 | """OpenCDN Node manager"""
19 | def __init__(self, arg):
20 | super(Node, self).__init__()
21 | self.arg = arg
22 |
23 | class Reload(object):
24 | """docstring for Reload"""
25 | def __init__(self, arg):
26 | super(Reload, self).__init__()
27 | self.arg = arg
28 | nginx_reload_queue.init()
29 |
30 | def nginx_reload_loop():
31 | while True:
32 | current_task_json_str = Queue.get_item_from_queue('OCDN_RELOAD')
33 | jobjson = JobJSON(current_task_json)
34 | (call_module, **arg) = jobjson.get_current_task_to_run()
35 | if self.nginx_reload(**arg) == False:
36 | self.nginx_reload_faliure()
37 | return False
38 |
39 | jobjson.set_next_task_to_run(parameters)
40 | Queue.put_item_into_queue('OCDN_RELOAD')
41 |
42 |
43 | def nginx_reload():
44 | """reolad nginx"""
45 | pass
46 |
47 | def nginx_reload_faliure():
48 | """reolad nginx failed"""
49 | pass
--------------------------------------------------------------------------------
/lib/purge.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python
2 | # -*- coding: UTF-8 -*-
3 |
4 | # author : firefoxbug
5 | # E-Mail : wanghuafire@gmail.com
6 | # Blog : www.firefoxbug.com
7 |
8 | """ OpenCDN Purge module: purge cdn cache.
9 |
10 | OpenCDN module instruction :
11 | OCDN_PURGE
12 |
13 | Purge API :
14 | http://node_ip:node_port/ocdn/purge/purge?token=node_token&domain=node_domain
15 | """
16 |
17 | import sys
18 | import os
19 | import time
20 | try:
21 | import json
22 | except Exception, e:
23 | import simplejson as json
24 |
25 | parent, bindir = os.path.split(os.path.dirname(os.path.abspath(sys.argv[0])))
26 | if os.path.exists(os.path.join(parent, 'lib')):
27 | sys.path.insert(0, os.path.join(parent, 'lib'))
28 |
29 | from OcdnQueue import Consumer
30 | from OcdnQueue import Producer
31 | from OcdnLogger import init_logger
32 | from OcdnJob import JobManager
33 | from OcdnTask import RunTask
34 |
35 | class Purge(Consumer):
36 | """OpenCDN Domain cache Purge
37 |
38 | 1. Register task module into Consumer
39 | 2. Start work loop get task from queue
40 | 3. Do task if failed redo it until exceed MaxRunTimes
41 | 4. If task done succuss and job is finished then update mysql
42 | 4. If task done success but job is unfinished then put next running task into queue
43 | """
44 | def __init__(self, queue_ip='103.6.222.21', queue_port=4730):
45 | self.queue_ip = queue_ip
46 | self.queue_port = queue_port
47 | self.CURRENT_TASK_MODULE = 'OCDN_PURGE'
48 | self.logfile = os.path.join(parent,'logs','%s.log'%(self.CURRENT_TASK_MODULE))
49 | super(Purge, self).__init__(queue_ip, queue_port, logfile=self.logfile)
50 | self.logger = init_logger(logfile=self.logfile, logmodule=self.CURRENT_TASK_MODULE, stdout=True)
51 | self.tasker = RunTask()
52 | self.taskid = None
53 |
54 | def run(self):
55 | """Register callback module and start a worker loop do tasks"""
56 | self.register_task_callback(self.CURRENT_TASK_MODULE, self.do_task)
57 | self.start_worker()
58 |
59 | def do_task(self, gearman_worker, job):
60 | """run a task may cause 3 results
61 |
62 | 1. task excuted failed
63 | 2. task excuted success and the job fished
64 | 3. task excuted success but the job unfished
65 | """
66 | data = job.data
67 | current_job_json = json.loads(data)
68 | jobmanager = JobManager(current_job_json)
69 | self.taskid = jobmanager.get_task_id()
70 | if not jobmanager.check_job_json_is_ok() :
71 | self.logger.error('TaskID:%s Parse job json failed. DATA:%s'%(self.taskid, job.data))
72 | return "False"
73 |
74 | # Get task and parameters to run
75 | task_name, task_args = jobmanager.get_current_task_to_run()
76 |
77 | # Run task
78 | if self.purge_node(task_args) == False:
79 | self.logger.error('TaskID:%s do task failed.'%(self.taskid))
80 | next_task_json = jobmanager.try_run_current_task_again()
81 | self.purge_node_failure(next_task_json)
82 | return "False"
83 |
84 | self.logger.info('TaskID:%s do task success'%(self.taskid))
85 | # Job is finished
86 | if jobmanager.is_job_finished():
87 | self.logger.info('TaskID:%s Job is finished'%(self.taskid))
88 | self.purge_job_success()
89 | return "True"
90 |
91 | # Job still has tasks to dispatch
92 | next_task_json = jobmanager.set_next_task_to_run()
93 | if not next_task_json :
94 | self.logger.error('TaskID:%s:[FAILED] Job is unfished but no more task to do'%(self.taskid))
95 | return "False"
96 | self.put_task_into_queue(next_task_json['CurrentTask'], next_task_json)
97 | return "True"
98 |
99 | def purge_node(self, task_args):
100 | """run task purge one node cache
101 |
102 | return False: job filed
103 | return True: job success
104 | """
105 | instance_list = []
106 | for item in task_args:
107 | purge_url = 'http://%s:%s/ocdn/purge/purge?token=%s&domain=%s'%(item['ip'], item['port'], item['token'], item['domain'])
108 | instance_list.append(purge_url)
109 | result = self.tasker.run(instance_list)
110 | # print result
111 | return False
112 |
113 | def purge_node_failure(self, next_task_json):
114 | """do with purge one node cache failured, try to dispatch the task again."""
115 | if not next_task_json :
116 | error_msg = 'TaskID:%s:[FAILED] Exceed MaxRunTimes, no more task to dispatch'%(self.taskid)
117 | self.logger.error(error_msg)
118 | else :
119 | info_msg = 'TaskID:%s try to redo task. AlreadyRunTimes=%s'%(self.taskid, next_task_json['RunTimesLimit']['AlreadyRunTimes'])
120 | self.logger.info(info_msg)
121 | self.put_task_into_queue(next_task_json['CurrentTask'], next_task_json)
122 |
123 | def purge_job_success(self):
124 | """The purge job is excuted successfully"""
125 | pass
126 |
127 | def put_task_into_queue(self, queue_name, task_json):
128 | """Put a new task json into task queue"""
129 | self.push_task(self.queue_ip, self.queue_port, str(queue_name), task_json)
130 |
131 | if __name__ == '__main__':
132 | purge = Purge()
133 | purge.run()
--------------------------------------------------------------------------------
/test/consumer.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python
2 | # -*- coding: UTF-8 -*-
3 |
4 | # author : firefoxbug
5 | # E-Mail : wanghuafire@gmail.com
6 | # Blog : www.firefoxbug.com
7 |
8 | """Test for opencdn queue
9 |
10 | consumer: get task from gearman server and do task
11 | """
12 | import sys
13 | import os
14 | import time
15 | import json
16 |
17 | parent, bindir = os.path.split(os.path.dirname(os.path.abspath(sys.argv[0])))
18 | if os.path.exists(os.path.join(parent, 'lib')):
19 | sys.path.insert(0, os.path.join(parent, 'lib'))
20 |
21 | from OcdnQueue import Consumer
22 | from OcdnLogger import init_logger
23 |
24 | class ConsumerTest(Consumer):
25 | """docstring for Consumer"""
26 | def __init__(self, queue_ip='103.6.222.21', queue_port=4730):
27 | self.task_name = 'CONSUMER'
28 | self.logfile = os.path.join(parent,'logs','%s.log'%(self.task_name))
29 | super(ConsumerTest, self).__init__(queue_ip, queue_port, logfile=self.logfile)
30 | self.logger = init_logger(logfile=self.logfile, logmodule='CONSUMER_TEST', stdout=True)
31 |
32 | def run(self):
33 | self.logger.info("Start work")
34 | self.register_task_callback('OCDN_PURGE', self.do_task)
35 | self.start_worker()
36 |
37 | def do_task(self, gearman_worker, job):
38 | data = job.data
39 | parameters = json.loads(data)
40 | Consumer.push_task(queue_ip='103.6.222.21', queue_port=4730, queue_name='OCDNQUEUE', data=parameters)
41 | print parameters
42 | return 'True'
43 |
44 | if __name__ == '__main__':
45 | consumer = ConsumerTest()
46 | consumer.run()
--------------------------------------------------------------------------------
/test/producer.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python
2 | # -*- coding: UTF-8 -*-
3 |
4 | # author : firefoxbug
5 | # E-Mail : wanghuafire@gmail.com
6 | # Blog : www.firefoxbug.com
7 |
8 | """Test for opencdn queue
9 |
10 | produce task and put task into gearman server
11 | """
12 | import sys
13 | import os
14 | parent, bindir = os.path.split(os.path.dirname(os.path.abspath(sys.argv[0])))
15 | if os.path.exists(os.path.join(parent, 'lib')):
16 | sys.path.insert(0, os.path.join(parent, 'lib'))
17 |
18 | from OcdnQueue import Producer
19 | from OcdnQueue import Consumer
20 | from OcdnLogger import init_logger
21 | from OcdnJob import OcdnJSON
22 |
23 | import pprint
24 | pp = pprint.PrettyPrinter()
25 |
26 | class ProducerTest(Producer):
27 | """docstring for queue_test"""
28 | def __init__(self, queue_ip='103.6.222.21', queue_port=4730):
29 | super(ProducerTest, self).__init__(queue_ip, queue_port)
30 | self.task_name = 'ProducerTest'
31 | self.logfile = os.path.join(parent,'logs','producer.log')
32 | self.logger = init_logger(logfile=self.logfile, stdout=True)
33 | self.logger.info('Connect to gearman server %s:%s'%(queue_ip, queue_port))
34 |
35 | def produce_job_loop(self, TaskModule, job2do):
36 | self.connect_queue()
37 | self.logger.info('Put task into TaskModule:%s'%(TaskModule))
38 | self.put_task_into_queue(TaskModule, job2do, Background=False)
39 |
40 | def produce_job_tes():
41 | Consumer.push_task(queue_ip='103.6.222.21', queue_port=4730, queue_name='OCDNQUEUE', data='hello')
42 |
43 | def purge_test():
44 | JobName = 'OCDN_PURGE'
45 | TaskList = ['OCDN_PURGE']
46 | Parameters = []
47 | Parameters.append({'ip':'192.168.1.1','port':'80','domain':'www.firefoxbug.com','token':'821e57c57e8455e3e809e23df7bb6ce9'})
48 | Parameters.append({'ip':'192.168.1.2','port':'80','domain':'www.firefoxbug.com','token':'821e57c57e8455e3e809e23df7bb6ce9'})
49 |
50 | test = OcdnJSON()
51 | job2do = test.create_job_json(JobName, TaskList, Parameters)
52 | pp.pprint(job2do)
53 |
54 | producer = ProducerTest()
55 | producer.produce_job_loop('OCDN_PURGE', job2do)
56 |
57 | def add_domain_test():
58 | TaskName = 'OCDN_PURGE'
59 | TaskList = ['OCDN_PURGE','OCDN_PROXY','OCDN_ADD_DNS']
60 | Parameters = []
61 | for _ in range(0,10) :
62 | Parameters.append({'ip':'192.168.1.1','port':'80','domain':'www.firefoxbug.com','token':'821e57c57e8455e3e809e23df7bb6ce9'})
63 | Parameters.append({'ip':'192.168.1.2','port':'80','domain':'www.firefoxbug.com','token':'821e57c57e8455e3e809e23df7bb6ce9'})
64 |
65 | test = OcdnJSON()
66 | job2do = test.create_job_json(TaskName, TaskList, Parameters)
67 | pp.pprint(job2do)
68 |
69 | producer = ProducerTest()
70 | producer.produce_job_loop(TaskName, job2do)
71 |
72 | if __name__ == '__main__':
73 | add_domain_test()
--------------------------------------------------------------------------------