├── .gitignore ├── Docs ├── Project Timeline.md ├── WSRTD Doc.md ├── Workflow Overview.md └── html download-light theme best view │ └── WSRTD Doc.md.html ├── LICENSE ├── Plugin └── notice.txt ├── README.md ├── Relay └── Server │ └── Python │ ├── notice.txt │ └── relay.py ├── Samples ├── Client │ └── Python │ │ ├── client.py │ │ ├── notice.txt │ │ └── vendor_class_wrapper.py └── Server │ └── Python │ ├── notice.txt │ └── sample_server.py ├── Utility Scripts └── Python │ ├── json_validator.py │ └── notice.txt ├── google04cf242d3a5a7326.html └── images ├── build.png ├── help ├── DB_Create.png ├── Plugin_Configure.png ├── Plugin_baloon_OK.png ├── Plugin_menu.png ├── RT_Data_index.png ├── Retrieve.png ├── TJP_Registry.png ├── debug1.png ├── sample_s1.png ├── sample_s2.png ├── workflow_1.png └── workflow_2.png └── unknown.svg /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # C extensions 7 | *.so 8 | 9 | # Distribution / packaging 10 | .Python 11 | build/ 12 | develop-eggs/ 13 | dist/ 14 | downloads/ 15 | eggs/ 16 | .eggs/ 17 | lib/ 18 | lib64/ 19 | parts/ 20 | sdist/ 21 | var/ 22 | wheels/ 23 | share/python-wheels/ 24 | *.egg-info/ 25 | .installed.cfg 26 | *.egg 27 | MANIFEST 28 | 29 | # PyInstaller 30 | # Usually these files are written by a python script from a template 31 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 32 | *.manifest 33 | *.spec 34 | 35 | # Installer logs 36 | pip-log.txt 37 | pip-delete-this-directory.txt 38 | 39 | # Unit test / coverage reports 40 | htmlcov/ 41 | .tox/ 42 | .nox/ 43 | .coverage 44 | .coverage.* 45 | .cache 46 | nosetests.xml 47 | coverage.xml 48 | *.cover 49 | *.py,cover 50 | .hypothesis/ 51 | .pytest_cache/ 52 | cover/ 53 | 54 | # Translations 55 | *.mo 56 | *.pot 57 | 58 | # Django stuff: 59 | *.log 60 | local_settings.py 61 | db.sqlite3 62 | db.sqlite3-journal 63 | 64 | # Flask stuff: 65 | instance/ 66 | .webassets-cache 67 | 68 | # Scrapy stuff: 69 | .scrapy 70 | 71 | # Sphinx documentation 72 | docs/_build/ 73 | 74 | # PyBuilder 75 | .pybuilder/ 76 | target/ 77 | 78 | # Jupyter Notebook 79 | .ipynb_checkpoints 80 | 81 | # IPython 82 | profile_default/ 83 | ipython_config.py 84 | 85 | # pyenv 86 | # For a library or package, you might want to ignore these files since the code is 87 | # intended to run in multiple environments; otherwise, check them in: 88 | # .python-version 89 | 90 | # pipenv 91 | # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. 92 | # However, in case of collaboration, if having platform-specific dependencies or dependencies 93 | # having no cross-platform support, pipenv may install dependencies that don't work, or not 94 | # install all needed dependencies. 95 | #Pipfile.lock 96 | 97 | # poetry 98 | # Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control. 99 | # This is especially recommended for binary packages to ensure reproducibility, and is more 100 | # commonly ignored for libraries. 101 | # https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control 102 | #poetry.lock 103 | 104 | # pdm 105 | # Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control. 106 | #pdm.lock 107 | # pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it 108 | # in version control. 109 | # https://pdm.fming.dev/latest/usage/project/#working-with-version-control 110 | .pdm.toml 111 | .pdm-python 112 | .pdm-build/ 113 | 114 | # PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm 115 | __pypackages__/ 116 | 117 | # Celery stuff 118 | celerybeat-schedule 119 | celerybeat.pid 120 | 121 | # SageMath parsed files 122 | *.sage.py 123 | 124 | # Environments 125 | .env 126 | .venv 127 | env/ 128 | venv/ 129 | ENV/ 130 | env.bak/ 131 | venv.bak/ 132 | 133 | # Spyder project settings 134 | .spyderproject 135 | .spyproject 136 | 137 | # Rope project settings 138 | .ropeproject 139 | 140 | # mkdocs documentation 141 | /site 142 | 143 | # mypy 144 | .mypy_cache/ 145 | .dmypy.json 146 | dmypy.json 147 | 148 | # Pyre type checker 149 | .pyre/ 150 | 151 | # pytype static type analyzer 152 | .pytype/ 153 | 154 | # Cython debug symbols 155 | cython_debug/ 156 | 157 | # PyCharm 158 | # JetBrains specific template is maintained in a separate JetBrains.gitignore that can 159 | # be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore 160 | # and can be added to the global gitignore or merged into this file. For a more nuclear 161 | # option (not recommended) you can uncomment the following to ignore the entire idea folder. 162 | #.idea/ 163 | -------------------------------------------------------------------------------- /Docs/Project Timeline.md: -------------------------------------------------------------------------------- 1 | # WS_RTD_AB 2 | #### _AmiBroker Realtime data plugin using Websocket and JSON based communication_ 3 | 4 | ## Project Timeline 5 | 6 | ##### 20241006 - Project Conceptualized 7 | https://forum.amibroker.com/t/data-plugin-design-discussions/39303 8 | ##### 20250101 - Added first Doc specs 9 | https://github.com/ideepcoder/Rtd_Ws_AB_plugin/blob/main/Docs/WSRTD%20Doc.md 10 | ##### 20250101 - Added HTML Download version, better view vs MD file 11 | https://github.com/ideepcoder/Rtd_Ws_AB_plugin/tree/main/Docs/html%20download-light%20theme%20best%20view 12 | ##### 20250109 - Opened WsRTD.dll for Alpha Test Release ( minimum 1-min base time ) 13 | https://forum.amibroker.com/t/wsrtd-websocket-json-standard-data-plugin-test-release/39640 14 | ##### 20250110 - Added High Level Workflow 15 | https://github.com/ideepcoder/Rtd_Ws_AB_plugin/blob/main/Docs/Workflow%20Overview.md 16 | ##### 20250113 - Added Python Sample Server 17 | https://github.com/ideepcoder/Rtd_Ws_AB_plugin/blob/main/Samples/Server/Python/sample_server.py 18 | ##### 20250113 - Added Python Sample Client 19 | https://github.com/ideepcoder/Rtd_Ws_AB_plugin/blob/main/Samples/Client/Python/client.py 20 | ##### 20250113 - Added Python Relay Server 21 | https://github.com/ideepcoder/Rtd_Ws_AB_plugin/blob/main/Relay/Server/Python/relay.py 22 | ##### 20250115 - Updated Python Sample Client - true async design 23 | https://github.com/ideepcoder/Rtd_Ws_AB_plugin/blob/main/Samples/Client/Python/client.py 24 | ##### 20250119 - Added Python General Class Wrapper for Vendor/Broker Library 25 | https://github.com/ideepcoder/Rtd_Ws_AB_plugin/blob/main/Samples/Client/Python/vendor_class_wrapper.py 26 | ##### 20250128 - Added Python utility script to Validate RTD Json 27 | https://github.com/ideepcoder/Rtd_Ws_AB_plugin/blob/main/Utility%20Scripts/Python/json_validator.py 28 | ##### 20250201 - Added Ping and AFL status function 29 | https://github.com/ideepcoder/Rtd_Ws_AB_plugin/blob/main/Docs/WSRTD%20Doc.md#afl-access-functions 30 | -------------------------------------------------------------------------------- /Docs/WSRTD Doc.md: -------------------------------------------------------------------------------- 1 | # WS_RTD_AB 2 | ## _AmiBroker Realtime data plugin using Websocket and JSON based communication_ 3 | 4 | [![Build Status](https://raw.githubusercontent.com/ideepcoder/Rtd_Ws_AB_plugin/84c47468847d2bbf53d2f12fa110d13c041d7b2d/images/unknown.svg)](https://github.com/ideepcoder/Rtd_Ws_AB_plugin) 5 | Doc version: 1.o, Plugin: 30014 6 | ## Features 7 | - Bi-directional websocket communication 8 | - Support to backfill historical data 9 | - Easy to integrate Json message structure 10 | - Run multiple instances of AmiBroker locally, each connected to one of multiple Servers 11 | - The design makes it Broker / Data vendor agnostic 12 | - Runs a python based relay server, to ensure individual connections don't disrupt your flow 13 | - Build your own data routing logic 14 | - Store your time-series data in the superfast Amibroker storage database 15 | - Get you Realtime Quote window ticking as opposed to plain ASCII imports 16 | 17 | # ✨ Upcoming ✨ 18 | - ArcticDB active integration 19 | - A lot of leverage with keeping bloated data locally in ArcticDB and a concise Amibroker database 20 | 21 | 22 | ### To-do: Table of Contents and Anchor links 23 | 24 | 25 | ##### For DLL/Binary redistribution, kindly read the LICENSE section. 26 | ============================================ 27 | 28 | ## Installation 29 | 30 | WS_RTD requires [AmiBroker] to run. 31 | | Software | Supported / Tested version | 32 | | ------ | ------ | 33 | | AmiBroker | v6.30+ x64-bit | 34 | | Visual C++ | VC++ 2022 redistributable | 35 | | Windows OS | Windows 10/11 x64-bit| 36 | | Linux OS| To-Do| 37 | | Vanilla Python | [3,12 x64-bit](https://www.python.org/downloads/release/python-3120/)| 38 | Install the above dependencies. 39 | Sample codes like sample_server.py requires additional python packages to be installed. 40 | 41 | ##### The plugin 42 | ```sh 43 | Copy WsRTD.dll to C:\PATH\AmiBroker\Plugins folder 44 | ``` 45 | 46 | ##### Run Amibroker and Create New Database 47 | https://www.amibroker.com/guide/w_dbsettings.html 48 | ```sh 49 | File > New > Database 50 | ``` 51 | 52 | ##### Configure the Database 53 | ```sh 54 | select Database folder 55 | Datasource: WsRtd data Plug-in 56 | Local data storage: ENABLE (very important) 57 | set: Number of bars 58 | Base time interval: 1 minute or as suited 59 | ``` 60 | *To-Do: Enabling seconds(timeframe) in next iteration. Tick-data unsupported for now* 61 | 62 | [![img create](https://github.com/ideepcoder/Rtd_Ws_AB_plugin/blob/main/images/help/DB_Create.png?raw=true)] 63 | 64 | ##### Configure the Plug-in 65 | ```sh 66 | Click Configure button 67 | ``` 68 | 69 | [![img configure](https://github.com/ideepcoder/Rtd_Ws_AB_plugin/blob/main/images/help/Plugin_Configure.png?raw=true)] 70 | 71 | The internal plug-in DB stores RTD Quotes as a RING BUFFER. Max_Num_Sym_Quotes here means the number of bars to hold. If you run Analysis frequently and ensure all symbols are accessed, then you can set it as low as 10 to reduce memory usage. Each bar is 40 KB. 72 | 73 | #### Important Settings change: 74 | 1. **Max_Num_SYM_Quotes** : Requires AB restart 75 | 2. ***Changes in IP / Port / AUTH** : Requires Plugin shutdown / connect, ie. Reconnect. 76 | 77 | ##### The Sample Server 78 | ensure these Python packages are installed and their dependencies 79 | ```sh 80 | import asyncio 81 | import websockets 82 | import datetime 83 | import json 84 | import random 85 | import sys 86 | import pandas as pd 87 | import copy 88 | ``` 89 | start the sample server in a Windows terminal, 90 | ```sh 91 | cd \path\to\server 92 | py sample_Server.py 93 | ``` 94 | [![img sample_server1](https://github.com/ideepcoder/Rtd_Ws_AB_plugin/blob/main/images/help/sample_s1.png?raw=true)] 95 | 96 | [![img sample_server1](https://github.com/ideepcoder/Rtd_Ws_AB_plugin/blob/main/images/help/sample_s2.png?raw=true)] 97 | 98 | **When Data Plug-in connects to the server** 99 | 100 | [![img baloon1](https://github.com/ideepcoder/Rtd_Ws_AB_plugin/blob/main/images/help/Plugin_baloon_OK.png?raw=true) 101 | 102 | 103 | # JSON FORMATS 104 | There are 4 types: Json-RTD, Json-HIST, Json-CMD, Json-ACK 105 | 106 | ### 1) RTD format in json message 107 | Currently, minimum periodicty/timeframe/interval of bars is 1 minute. 108 | Valid fields in the json object 109 | **ORDER** See the Json recommended section, rtd message has to start with "n" field. Subsequent order does not matter. 110 | **STRUCTURE** This is Array of JSON strings 111 | 112 | >'n', 'd', 't', 'o', 'h', 'l', 'c', 'v', 'oi', 'x1', 'x2', ( for Quotations ) 113 | 's','pc','bs','bp','as','ap', 'do', 'dh', 'dl' ( for Realtime Quote window) 114 | 115 | > 'n'=symbol name, date, time, open, high, low, close, volume, open interest, 116 | >'x1'=Aux1, 'x2'=Aux2 117 | 's'=total daily vol, 'pc'=prev day close, 118 | 'do', 'dh', 'dl', are Day Open, High, Low 119 | 'bs', 'bp', 'as', 'ap', are bid size/ bid price /ask size / ask price 120 | 121 | >Sample rtd json 122 | ```sh 123 | [{"n":"SYM1","t":101500,"d":20241130,"c":3,"o":6,"h":9,"l":1,"v":256,"oi":0,"bp":4,"ap":5,"s":12045,"bs":1,"as":1,"pc":3,"do":3,"dh":9,"dl":1},{"n":"SYM2","t":101500,"d":20241130,"c":3,"o":6,"h":9,"l":1,"v":256,"oi":0,"bp":4,"ap":5,"s":12045,"bs":1,"as":1,"pc":3,"do":3,"dh":9,"dl":1},{"n":"SYM3","t":101500,"d":20241130,"c":3,"o":6,"h":9,"l":1,"v":256,"oi":0,"bp":4,"ap":5,"s":12045,"bs":1,"as":1,"pc":3,"do":3,"dh":9,"dl":1}] 124 | ``` 125 | Supported Date/Time formats: 126 | ```sh 127 | milliseconds not yet supprted 128 | u = unix timestamp localtime() ( type: integer64 ) 129 | g = unix timestamp gmtime() ( type: integer64 ) 130 | d = date ( type: integer ) 131 | t = time ( type: integer ) 132 | u, g and d are MUTUALLY EXCLUSIVE, but any one is mandatory. 133 | "t" is mandatory for RTD along with "d" 134 | /U=unix_local_ms /G=unix_utc_ms (To-Do:) 135 | ``` 136 | 137 | Refer to Amibroker ADK link above for data types. 138 | 'd'.'t','as','bs' as integer, 139 | float is safe for the rest. 140 | To-do: unix timestamp(u/g), seconds interval 141 | 142 | if required fields are not found, they are set to ZERO & that will distort candles. 143 | ```sh 144 | o, h, l, c = open,high,low,last_traded_price = mandatory for new Quote 145 | v, oi, x1, x2 = optional and set to 0 if absent 146 | 147 | Realtime Quote window variables are optional, but they 148 | are required for this feature to work properly. 149 | ``` 150 | Using d,t is more efficient over u,g as 1 step less in conversion and should be preferred if available. 151 | 152 | ### 2) History format in json message 153 | Parse a history data JSON string that is used to backfill a Ticker 154 | 155 | >This is all the backfill records in Array format for a single ticker 156 | format specifier is fieldsStr = "dtohlcvixy" 157 | only valid fields should be set and in that particular order. 158 | 159 | **ORDER** The sequence of columns in hostorical data can be anything set by the user, but it has to match its character in the Format specifier string. The beginning of the json message will start with "hist" containing Symbol name followed by "format" field containing the Format specifier string. 160 | 161 | **Important: History Array records must be in ascending order**, ie. Least DateTime first. 162 | ```sh 163 | (ugdt)ohlcvixy 164 | milliseconds not yet supprted 165 | u = unix timestamp localtime() ( type: integer64 ) 166 | g = unix timestamp gmtime() ( type: integer64 ) 167 | d = date ( type: integer ) 168 | t = time ( type: integer ) 169 | u, g and d are MUTUALLY EXCLUSIVE, but any one is mandatory. 170 | "t" is mandatory for intraday bars with "d" 171 | 172 | o = open price ( type: float ), required 173 | h = high price ( type: float ), required 174 | l = low price ( type: float ), required 175 | c = close price ( type: float ), required 176 | v = volume ( type: float ), optional 177 | i = open interest ( type: float ), optional 178 | x = aux1 ( type: float ), optional 179 | y = aux2 ( type: float ), optional 180 | /U=unix_local_ms /G=unix_utc_ms (To-Do:) 181 | ``` 182 | if required fields are not found, they are set to ZERO & that will distort candles. 183 | 184 | ##### bars field is JSON Value Array 185 | >Sample Date-Time 186 | >DATE (optional TIME) d,t is supplied as INT, 187 | ```sh 188 | {"hist":"SYM1","format":"dtohlcvi","bars":[[20240601,100000,22.1,22.5,22.1,22.8,120,0], 189 | [20240602,110000,22.8,22.7,22.3,22.4,180,0],[20240603,120000,22.8,22.7,22.3,22.4,180,0]]} 190 | ``` 191 | 192 | >Sample unix timestamp 193 | ```sh 194 | ({"hist":"SYM1","format":"uohlcv","bars":[[1731670819,22.1,22.5,22.1,22.8,120], 195 | [1731761419,22.8,22.7,22.3,22.4,180]]} 196 | ``` 197 | >u=Unix Local timestamp OR g=Unix GMT/UTC timestamp 198 | that is use ( windows struct tm = localtime() or tm = gmtime() 199 | 200 | 201 | ##### Backfill style-01 202 | >Currently, Vendor backfill means N most recent days data 203 | so plugin also sets Bars from the first of this Array to the last bar 204 | the existing bars are overwritten, from first bar of hist. 205 | old_data + WS_data 206 | Now, backfill means old_data + backfill (upto last backfill) 207 | then, old_data + backfill_data + Rt_data ( from bars after last backfill bar, if any ) 208 | 209 | [![img sample_server1](https://github.com/ideepcoder/Rtd_Ws_AB_plugin/blob/main/images/help/Plugin_menu.png?raw=true) 210 | 211 | ##### Backfill style-02 *To-do:* 212 | Inserting historical data into AmiBroker Database 213 | 214 | ### 3) CMD format in json message 215 | Work in progress, subject to change. 216 | > ALL REQUESTS made by **plug-in TO** Server have only "cmd" and "arg fields" 217 | > Server should REPLY to all above REQUESTS with same "cmd" and "arg" fields. Additionally inserting the "code" which is integer. 218 | > Code values for OK=200 to 299, and OTHER/BAD=400 to 499. Specific codes are not defined yet. 219 | 220 | > ALL REQUESTS made by **Server To** Plug-in will have valid "cmd" and "arg" fields. Additionally, all SERVER-side requests will have code=300 which is integer. 221 | >( The only exceptions is json-RTD and json-HIST formats.) 222 | 223 | > Plug-in will respond with **Acknowledgement TO Server** for all code=300 Server-side REQUESTS with json-ACK "ack" with server-request, "arg" with description and "code" which is integer 224 | > Code values for OK=200 to 299, and OTHER/BAD=400 to 499. Specific codes are not interpreted currently. 225 | 226 | 227 | #### 3.1) Request CMD from plugin to "Server" 228 | 229 | ##### a) First Access automatic symbol backfill request ( use as SUBSCRIBE signal ) 230 | This is a special automatic backfill request command that is sent to the server when a symbol is accessed in Amibroker for the first time. Server can choose to ignore this request or respond with a json-History backfill data of a desired period. 231 | {"cmd":"bfauto","arg":"SYMBOL_NAME DateNum TimeNum"} last Quote timestamp sent when symbol has some data 232 | {"cmd":"bffull","arg":"SYMBOL_NAME"} sent when symbol has NO data 233 | ```sh 234 | {"cmd":"bfauto","arg":"SYM3 20241125 134500"} 235 | {"cmd":"bffull","arg":"SYM9"} 236 | ``` 237 | 238 | ##### b) single symbol backfill 239 | {"cmd":"bfsym","arg":"reserved SYMBOL_NAME int_preset"} 240 | ```sh 241 | {"cmd":"bfsym","arg":"y SYM1 3"} 242 | ``` 243 | 244 | ##### c) ALL symbol backfill 245 | {"cmd":"bfall","arg":"x"} 246 | ```sh 247 | {"cmd":"bfall","arg":"x"} 248 | ``` 249 | For ALL Symbol backfill, client application should still send individual Json-hist messages for each symbol. 250 | 251 | ##### d) ADD symbol to SUBSCRIBE rtd 252 | {"cmd":"addsym","arg":"SYMBOL_NAME"} 253 | ```sh 254 | {"cmd":"addsym","arg":"SYM10"} 255 | ``` 256 | 257 | ##### e) REMOVE symbol OR UNSUBSCRIBE rtd 258 | {"cmd":"remsym","arg":"SYMBOL_NAME"} 259 | ```sh 260 | {"cmd":"remsym","arg":"SYM6"} 261 | ``` 262 | 263 | ##### f) CLIENT-APP connected request ping 264 | {"cmd":"cping","arg":""} 265 | ```sh 266 | {"cmd":"cping","arg":""} 267 | ``` 268 | Get state if any Client-App is connected to the relay. Also, a ping-alive packet that wsrtd is connected to Relay server. 269 | 270 | 271 | #### 3.2) Response CMD "to WS_RTD" Plug-in from Server 272 | 273 | ##### a) General acknowledgement response 274 | {"cmd":"CMD_SENT","code":int_code,"arg":"response string"} 275 | Mandatory code field 276 | ```sh 277 | {"cmd":"remsym","code":200,"arg":"SYM6 unsubscribed ok"} /* sucess example*/ 278 | {"cmd":"addsym","code":400,"arg":"XYZ9 subscribe error, invalid"} /* failure example*/ 279 | ``` 280 | 281 | ##### b) Ping acknowledgement response 282 | Mandatory code field 283 | ```sh 284 | {"cmd":"cping","code":200,"arg":"Vendor Connected"} /* Client running & connected to remote*/ 285 | {"cmd":"cping","code":400,"arg":"Vendor Disconnected"} /* Client running but remote source is disconnected*/ 286 | ``` 287 | 288 | 289 | #### 3.3) Request CMD "to WS_RTD" Plug-in "from" Server 290 | Mandatory code=300 291 | {"cmd":"request_cmd","code":300,"arg":"request_arg"} 292 | 293 | ##### a) REMOVE or DELETE a Symbol "IN" plug-in Database 294 | {"cmd":"dbremsym","code":300,"arg":"SYMBOL_NAME"} 295 | Case-sensitive match 296 | This is useful to FREE memory in the plug-in DB 297 | ```sh 298 | {"cmd":"dbremsym","code":300,"arg":"SYM9"} 299 | returns {"ack","code":200."arg":"SYM9 removed from DB"} /* success */ 300 | returns {"ack","code":400,"arg":"SYM9 NOT found in DB") /* failure */ 301 | ``` 302 | 303 | ##### b) Get the List of Symbols in Plug-in Database 304 | {"cmd":"dbgetlist","code":300,"arg":""} 305 | "arg" field required, can set empty or some string description 306 | returns a comma-string of symbols 307 | ```sh 308 | {"cmd":"dbgetlist","code":300,"arg":"DB Symbol list requested at 14:56:00"} 309 | returns {"ack":"dbgetlist","code":200,"arg"="AA,AAPL,AXP,BA,C,CAT,IBM,INTC,"} 310 | ``` 311 | 312 | ##### c) Get the Base Time Interval of Plug-in Database 313 | {"cmd":"dbgetbase","code":300,"arg":""} 314 | "arg" field required, can set empty 315 | returns a STRING with base time interval in seconds(convert to INT) 316 | ```sh 317 | {"cmd":"dbgetbase","code":300,"arg":"DB Symbol list requested at 14:56:00"} 318 | returns {"ack":"dbgetbase","code":200,"arg"="60"} 319 | ``` 320 | 321 | ##### d) Get the DB Status of Plug-in Database 322 | {"cmd":"dbstatus","code":300,"arg":""} 323 | "arg" field required, can set empty 324 | returns a STRING with Format as ( convert to INT ) 325 | Current_Sym_Count, Max_Sym_Limit, Max_Size_Sym_Quotes, Refresh_Interval, Base_Interval, DB_Name 326 | ```sh 327 | {"cmd":"dbstatus","code":300,"arg":"DB Status requested at 14:56:00"} 328 | returns {"ack":"dbstatus","code":200,"arg"="500 1000 200 300 60 Db_Test"} 329 | ``` 330 | 331 | 332 | ### 4) ACK format in json message 333 | Plug-in will respond with **Acknowledgement TO Server** for all code=300 Server-side REQUESTS with json-ACK "ack" with server-request, "arg" with description and "code" which is integer 334 | Code values for OK=200, and OTHER/BAD=400 335 | ```sh 336 | For samples, check 3.3 Plug-in return messages 337 | ``` 338 | 339 | 340 | ### Important: Json string compression and format 341 | * Json message should be compressed removing all whitespaces or prettify. 342 | * Use your library Json Encoder to prevent errors. 343 | For example, *The JSON standard requires double quotes and will not accept single quotes* 344 | * The json-type of messages are string matched at the start 345 | * Json CMD, history-format string and RTD fields are all **case-sensitive** 346 | ```sh 347 | C++ case-sensitive Raw string match example for performance 348 | R"([{"n")" // Realtime Data 349 | R"({"cmd")" // Commands 350 | R"({"hist")" // Historical Data 351 | ``` 352 | 353 | ## Database Plug-in Commands 354 | Explained here [Amibroker Forum](https://forum.amibroker.com/t/documentation-on-batch-data-plugin-command/39269) 355 | Amibroker [Batch window manual](https://www.amibroker.com/guide/h_batch.html) 356 | 357 | Every time the Plug-in receives data, it stores is internally in its own temporary storage and notifies Amibroker of new data. Unfortunately, by design, if the Symbol is not being used by the user in any way, the plug-in may never be able to "push" the data to AB local Database that is persistent. 358 | Therefore, user needs to use a Batch file that consists of an empty Analysis Exploration to force all symbols to fetch data. More on this later. 359 | 360 | These are some ways to communicate with the Plug-in from the Amibroker User-Interface OR when the websocket is shutdown and therefore not reachable externally. 361 | 362 | ##### 1) PushDBtoAB 363 | Batch command to inform plug-in that an all symbol fetch has been performed. 364 | 365 | ##### 2) ConnectPlugin 366 | Batch command to start the Websocket connection. 367 | 368 | ##### 3) ShutdownPlugin 369 | Batch command to stop the Websocket connection. 370 | 371 | ##### 4) DbStatus 372 | Batch command to log some Plug-in statistics. 373 | ```sh 374 | [1] DBG: Db() Status Map_size=4, Qt_RingBufr=200, SymList=SYM1,SYM3,SYM4,SYM5, 375 | ``` 376 | ##### 5) Clear First Access/Automatic backfill data 377 | For every first access of a symbol, bfauto or bffull request is sent once only. You can clear the flag with this. 378 | 379 | < Add more commands as required> 380 | 381 | ## Logging and troubleshooting 382 | #### 1) DebugView 383 | WS_RTD uses extensive logging using Windows Debug output. 384 | - Just run [DebugView](https://learn.microsoft.com/en-us/sysinternals/downloads/debugview) 385 | - Set the Filter to "DBG" *(may change this string later)* 386 | - View realtime Plug-in related information here 387 | 388 | [![img dbgview11](https://github.com/ideepcoder/Rtd_Ws_AB_plugin/blob/main/images/help/debug1.png?raw=true)] 389 | 390 | #### 2) One or some symbols not updating 391 | Check under Symbol > Information window, 392 | ```sh 393 | Use only local database = No (Ensure No, If yes, plug-in-cannot update it) 394 | ``` 395 | 396 | #### 3) View Count of bars that are in Plug-in DB and those that have been updated to Amibroker DB 397 | Whenever new data arrives, whether RTD or Historical, it is immediately stored internally. Amibroker is notified, but AB will choose to to request data only when that symbol is being used. 398 | To visualize that, currently, Two fields in the AB REaltime Quote window are dedicated to that. 399 | ```sh 400 | EPS: This column shows the Total count of bars in Plug-in DB 401 | ``` 402 | ```sh 403 | Dividend: This column shows the Total count of bars in updated to AB from the Plug-in DB 404 | ``` 405 | [![RT window1](https://github.com/ideepcoder/Rtd_Ws_AB_plugin/blob/main/images/help/RT_Data_index.png?raw=true)] 406 | 407 | #### 4) AB Plug-in UI Commands: Connect and Shutdown 408 | These commands will only Connect or Disconnect the plug-in (Client Websocket) to the local relay server. Data stored in the Plug-in DB is not affected. 409 | AB Symbols will still get updated if there is fresh data in the plug-in DB while the user has disconnected the plug-in. 410 | The plug-in DB is not persistent, that means, if user Exits Amibroker or Changes the Current DB in Amibroker, only then the plug-in data is cleared. 411 | 412 | #### 5) Sample output of Configure settings 413 | ```sh 414 | DBG: ConfDlg() RI=300ms, Conn=127.0.0.1:10101, SymLimit=1000, QuoteLimit=250, DbName=Data 415 | ``` 416 | < more items to come > 417 | 418 | 419 | # WS_RTD Plugin Settings storage 420 | The plugin stores settings in the Windows Registry like Amibroker QT.dll 421 | Amibroker path 422 | ```sh 423 | Computer\HKEY_CURRENT_USER\SOFTWARE\TJP\wsRtD 424 | ``` 425 | ```sh 426 | Nested in TJP\wsRtD\ 427 | ``` 428 | Settings will be UNIQUE to each unique database name created in AB. 429 | ALL Amibroker Databases that share common Database name but are different filesystem path will "still" share the same settings. 430 | This is the best of both, allows the user to separate or share settings as required. 431 | ```sh 432 | Example 1: 433 | Amibroker DB name: Data with their respective paths 434 | C:\sample1\Data 435 | D:\archive\old_sample\Data 436 | ``` 437 | *Both AB DBs above will share the same settings and there will be only one registry key named "Data"* 438 | ```sh 439 | Example 2: 440 | C:\sample1\Data_rtd 441 | C:\sample1\wsrtd 442 | ``` 443 | *Data2 and wsrtd will be a new entries and each can have its own settings* 444 | Image of Windows Registry below: 445 | 446 | [![img sample_server1](https://github.com/ideepcoder/Rtd_Ws_AB_plugin/blob/main/images/help/TJP_Registry.png?raw=true)] 447 | 448 | 449 | 450 | ## Client-Data Sender application 451 | For production release: 452 | ```sh 453 | To do: 454 | For now, this integration depends on the user and technical skill 455 | Client-sender program should be developed using Vendor specific library. 456 | Working on providing a Relay Server 457 | ``` 458 | 459 | > Note: `--rolesend` is required for client-sender identification 460 | 461 | Example rolesend: data vendor / broker client program / Python sample server 462 | 463 | > Note: `--rolerecv` is required for client-receiver identification 464 | 465 | Example rolerecv: Ws_Rtd plugin or ArcticDB data collection socket conn 466 | 467 | Verify the connection by navigating to your terminal running the server 468 | 469 | ```sh 470 | 127.0.0.1:10101 471 | ``` 472 | #### 1) Specifics of Client-Sender Application 473 | The design itself keeps in mind a minimal requirement so the application just needs to conform with the fol1owing: 474 | - Client-Sender connects to the matching websocket port 475 | - On connection, it sends the above simple string "rolesend" before any other data 476 | - Then it follows the json-RTD format for realtime bars matching the interval set in AB configure DB 477 | - To handle Requests from Data Plug-in, it will receive the json-CMD or json-Hist. It should respond with the appropriate json formats described above. That's it. 478 | - The Advantage of using a relay server is that when either Plug-ins or Client applications reconnect or disconnect, they don't cause the other party websocket drop errors which is distracting as well. 479 | 480 | < more documentation on this later > 481 | The Programming Language of choice or platform does not in any way become restrictive to the user. 482 | 483 | # FAQs 484 | #### 1) Can I Backfill data when the Status is Shutdown? 485 | As the websocket is disconnected, no commands can be sent or received, therefore it is not possible. 486 | However, Data available inside the plugin-DB can still be accessed for symbols to get updated. 487 | 488 | #### 2) I Imported data using Amibroker ASCII Importer, now I have duplicate or Corrupt data. 489 | By design, it is up to the user to ensure that duplicate data is not imported. Mixing two different methods can cause unwanted behaviour. 490 | You can delete all the data and import it afresh. 491 | Some steps are taken when using plug-in to overwrite duplicate data. 492 | However, this Plug-in is quite technical and for performance, the user's Client side application should ensure data integrity. 493 | 494 | #### 3) Some of my Symbols are not updating with data but my Server shows that data for RTD is generated. 495 | Ensure that under Symbol Information window in Amibroker, the setting "Use local database only" is set to No. If yes, Plug-in cannot update to it. 496 | 497 | #### 4) How do I ADD symbols in AmiBroker? 498 | When new symbols arrrive in the plugin-DB, the plugin status will be dark Green in colour. You can go to Configure settings Dialog and Click RETRIEVE Button. Symbols will automatically get added. 499 | 500 | [![Retrieve](https://raw.githubusercontent.com/ideepcoder/Rtd_Ws_AB_plugin/refs/heads/main/images/help/Retrieve.png)] 501 | 502 | The status color will change to 🟢 from $${\color{ForestGreen}Dark \space Green}$$ 503 | 504 | #### 5) Settings change in Configure 505 | Kindly use DebugView to check if settings change requires an AB to be restarted or Plugin to be reconnect. Scroll up to Configure section and read the details. 506 | 507 | 508 | ## AFL access functions 509 | ### using GetExtraData() 510 | https://www.amibroker.com/guide/afl/getextradata.html 511 | 512 | #### 1) "IsRtdConn" 513 | if Data plugin websocket is connected, 1 else 0 514 | 515 | #### 2) "ClientAppStatus" 516 | if Client App websocket is connected, Client App should implement "cping" cmd 517 | ```sh 518 | 0 = client-app not running 519 | 200 = running 520 | 400 = running but remote disconnected 521 | -1 = ping sent, awaiting reply 522 | ``` 523 | 524 | 525 | 526 | ## Development 527 | Want to contribute? Great! 528 | ##### A) 529 | For now Clent-sender applications have a lot of scope for expansion 530 | Current Relay Server is python-based. 531 | Client-sender can be in a Programming Language of your choice or using Data Vendor library, 532 | these applications can be expanded greatly to encompass various Brokers APIs and Data Vendors. 533 | ##### B) 534 | WS_RTD uses C++ for development, strongly based on ATL/MFC, but is currently closed. 535 | 536 | ## License 537 | **To be decided** 538 | 539 | This LICENSE Section alongwith the CREDIT and AUTHOR Section at minimum should be distributed EXACTLY as-is when distributing a copy of the data-plugin binary or DLL. 540 | 541 | ## Author 542 | NSM51 543 | A fan and avid Amibroker user. 544 | https://forum.amibroker.com/u/nsm51/summary 545 | < Journey story about the idea of this plugin > 546 | 547 | # Tech & Credits: 548 | ## AB Data Plugin 549 | Inspired by QT.dll sample from AmiBroker ADK 550 | https://gitlab.com/amibroker/adk/-/tree/master/Samples/QT 551 | Credits: [Amibroker] Company & Dr. Tomasz Janeczko 552 | 553 | ## Web Socket 554 | Current WS library: EasyWsClient 555 | https://github.com/dhbaird/easywsclient 556 | License: MIT 557 | Works well as light-weight windows WinSock2 wrapper. 558 | Credits: dhbaird 559 | 560 | ## JSON 561 | Current JSON library: RapidJson 562 | https://github.com/Tencent/rapidjson 563 | License: MIT 564 | Credits: Milo Yip 565 | RapidJson is benchmarked as fastest. 566 | 567 | ## WS Class and Queue inspiration 568 | https://stackoverflow.com/a/69054531, Credits: Jerry Coffin 569 | 570 | *Created using* [Dillinger] 571 | 572 | ###### Doc End 573 | [//]: # (These are reference links used in the body of this note and get stripped out when the markdown processor does its job. There is no need to format nicely because it shouldn't be seen. Thanks SO - http://stackoverflow.com/questions/4823468/store-comments-in-markdown-syntax) 574 | 575 | [Amibroker]: 576 | [Dillinger]: 577 | -------------------------------------------------------------------------------- /Docs/Workflow Overview.md: -------------------------------------------------------------------------------- 1 | # WSRTD Workflow 2 | ## High level implementation design 3 | 4 | ## 1) Data plugin interaction from ADK (AmiBroker) 5 | 6 | [![ADK Plugin](https://raw.githubusercontent.com/ideepcoder/Rtd_Ws_AB_plugin/refs/heads/main/images/help/workflow_1.png) 7 | 8 | ## 2) Local Websocket Server/Client Models 9 | 10 | [![WSRTD Plugin](https://raw.githubusercontent.com/ideepcoder/Rtd_Ws_AB_plugin/refs/heads/main/images/help/workflow_2.png) 11 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2024 ideepcoder 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /Plugin/notice.txt: -------------------------------------------------------------------------------- 1 | This plugin should be distributed or redisributed with the Documentation 2 | file provided in HTML that contains all the Licence information, 3 | Trademark, Copyright & Other notices. -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Rtd_Ws_AB_plugin 2 | ## _AmiBroker Realtime data plugin using Websocket and JSON based communication_ 3 | 4 | [![Build Status](https://raw.githubusercontent.com/ideepcoder/Rtd_Ws_AB_plugin/84c47468847d2bbf53d2f12fa110d13c041d7b2d/images/unknown.svg)](https://github.com/ideepcoder/Rtd_Ws_AB_plugin) 5 | 6 | 7 | ## Features 8 | - Bi-directional websocket communication 9 | - Support to backfill historical data 10 | - Easy to integrate Json message structure 11 | - Run multiple instances of AmiBroker locally, each connected to one of multiple Servers 12 | - The design makes it Broker / Data vendor agnostic 13 | - Runs a python based relay server, to ensure individual connections don't disrupt your flow 14 | - Build your own data routing logic 15 | - Store your time-series data in the superfast Amibroker storage database 16 | - Get you Realtime Quote window ticking as opposed to plain ASCII imports 17 | 18 | # ✨ Upcoming ✨ 19 | - ArcticDB active integration 20 | - A lot of leverage with keeping bloated data locally in ArcticDB and a concise Amibroker database 21 | 22 | 23 | ## Documentation and manual 24 | ### -- Check Docs for the latest documentation 25 | 26 | #### [Main Documentation](https://github.com/ideepcoder/Rtd_Ws_AB_plugin/blob/main/Docs/WSRTD%20Doc.md) 27 | #### [Project Timeline](https://github.com/ideepcoder/Rtd_Ws_AB_plugin/blob/main/Docs/Project%20Timeline.md) 28 | 29 | # . 30 | ## Licenses / Copyrights / Trademarks 31 | ##### This work incorporates materials copyrighted and trademarked by AmiBroker Company. 32 | AmiBroker and the AmiBroker logo are trademarks of AmiBroker Company, used with permission. 33 | 34 | 35 | -------------------------------------------------------------------------------- /Relay/Server/Python/notice.txt: -------------------------------------------------------------------------------- 1 | This plugin should be distributed or redisributed with the Documentation 2 | file provided in HTML that contains all the Licence information, 3 | Trademark, Copyright & Other notices. -------------------------------------------------------------------------------- /Relay/Server/Python/relay.py: -------------------------------------------------------------------------------- 1 | ''' 2 | // ##### Relay_Server.py ##### 3 | // 4 | // Independent Relay server for WsRtd AmiBroker Data Plugin 5 | // 6 | // Central Websocket Python Program that runs as a Server. 7 | // Both, CLient Application RTD Senders and the WsRTD data plugin 8 | // connect to this server via specified IP:Port 9 | // 10 | // The advantage is that either Sender-Client or Receiver-client 11 | // can restart or drop the connection without affecting the other. 12 | // In the future, it will also serve to hook ArcticDB integration into the system. 13 | // 14 | /////////////////////////////////////////////////////////////////////// 15 | // Author: NSM51 16 | // https://github.com/ideepcoder/Rtd_Ws_AB_plugin/ 17 | // https://forum.amibroker.com/u/nsm51/summary 18 | // 19 | // Users and possessors of this source code are hereby granted a nonexclusive, 20 | // royalty-free copyright license to use this code in individual and commercial software. 21 | // 22 | // AUTHOR ( NSM51 ) MAKES NO REPRESENTATION ABOUT THE SUITABILITY OF THIS SOURCE CODE FOR ANY PURPOSE. 23 | // IT IS PROVIDED "AS IS" WITHOUT EXPRESS OR IMPLIED WARRANTY OF ANY KIND. 24 | // AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOURCE CODE, 25 | // INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. 26 | // IN NO EVENT SHALL AUTHOR BE LIABLE FOR ANY SPECIAL, INDIRECT, INCIDENTAL, OR 27 | // CONSEQUENTIAL DAMAGES, OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, 28 | // WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, 29 | // ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOURCE CODE. 30 | // 31 | // Any use of this source code must include the above notice, 32 | // in the user documentation and internal comments to the code. 33 | ''' 34 | 35 | 36 | ## stable with ws-close(), handles with BaseException catch. Just prints the exception though. 37 | ## use as middle server to recv from one client and send to another. 38 | ## this is working code with ctrl+c issue fixed 39 | wsport = 10101 40 | 41 | import asyncio 42 | import websockets 43 | import threading 44 | import datetime 45 | import sys 46 | 47 | stop_event = 0 48 | stop_threads = False 49 | CLIENTS = set() 50 | SENDERS = set() 51 | ctr = [0, 0, 0] # counters: clients, senders, reserved 52 | retCode = 0 53 | 54 | 55 | ''' 56 | Function to iterate over all CLIENTS and broadcast message received from SENDER 57 | ''' 58 | async def broadcast_c( message): 59 | 60 | Bad_WS = set() 61 | 62 | for websocket in CLIENTS: 63 | try: 64 | await websocket.send( message ) 65 | except websockets.ConnectionClosed: Bad_WS.add( websocket ); continue 66 | except ConnectionResetError: Bad_WS.add( websocket ); continue 67 | except Exception as e: print(f"broadcast_C() {e}"); break 68 | 69 | if len( Bad_WS ) > 0: 70 | for ws in Bad_WS: 71 | CLIENTS.remove( ws ) 72 | 73 | 74 | 75 | ''' 76 | Function to iterate over all SENDER(S) and broadcast message received from CLIENTS 77 | ''' 78 | async def broadcast_s( message): 79 | 80 | Bad_WS = set() 81 | 82 | for websocket in SENDERS: 83 | try: 84 | await websocket.send( message ) 85 | except websockets.ConnectionClosed: Bad_WS.add( websocket ); continue 86 | except ConnectionResetError: Bad_WS.add( websocket ); continue 87 | except Exception as e: print(f"broadcast_S() {e}"); break 88 | 89 | 90 | if len( Bad_WS ) > 0: 91 | for ws in Bad_WS: 92 | SENDERS.remove( ws ) 93 | 94 | 95 | 96 | ''' 97 | Main function that creates Handler for Each websocket connection (ie. Send() / Receive() functionality) 98 | ''' 99 | async def handler(websocket): 100 | global stop_event, stop_threads, SENDERS, CLIENTS, ctr 101 | role = l_role = 0 ## local role 102 | 103 | try: 104 | role = await websocket.recv() 105 | 106 | if str(role).startswith('role'): 107 | if str(role).endswith('send'): SENDERS.add( websocket); l_role = 1 108 | else: CLIENTS.add( websocket); l_role = 2 109 | except: pass 110 | 111 | ## create periodic task: ## disabled task to work as echo server. Client (fws) -> WS_server->Client C++ 112 | #asyncio.create_task(send(websocket)) 113 | try: 114 | if l_role==1: 115 | print(f"sender conn"); ctr[1]+=1; await stats(); await asyncio.create_task( senders_t( websocket)) 116 | print(f"sender disc"); ctr[1]-=1 117 | elif l_role==2: 118 | print(f"client conn"); ctr[0]+=1; await stats(); await asyncio.create_task( clients_t( websocket)) 119 | print(f"client disc"); ctr[0]-=1 120 | else: print(f"Bad Auth: {role}"); await websocket.send('Server: Bad or No Auth') 121 | 122 | except TimeoutError: pass 123 | except websockets.ConnectionClosed: return 124 | except ConnectionResetError: return 125 | except Exception as e:print(f"handle() Ex {e}"); return 126 | except BaseException as e:print(f"handle() BEx {e}");return 127 | 128 | 129 | 130 | ''' 131 | Individual Role-Senders RECEIVE function as task, this message is BROADCAST to all CLIENTS 132 | ''' 133 | async def senders_t( websocket): 134 | global stop_threads, stop_event 135 | while not stop_threads: 136 | try: 137 | # this code is for normal recv but now above is just echo 138 | message = await websocket.recv() 139 | #print(message) 140 | await broadcast_c( message) 141 | 142 | except TimeoutError: pass 143 | except websockets.ConnectionClosed: break ## connection drops 144 | except ConnectionResetError: break 145 | except Exception as e:print(f"senders() Ex {e}"); break 146 | return 0 147 | 148 | 149 | 150 | ''' 151 | Individual Role-Clients RECEIVE function as task, this message is BROADCAST to all SENDERS 152 | ''' 153 | async def clients_t( websocket): 154 | global stop_threads, stop_event 155 | while not stop_threads: 156 | try: 157 | # this code is for normal recv, can use async as well for timeout 158 | message = await websocket.recv() 159 | print(message) # print client msessages in server 160 | await cmdMmessage( message) 161 | 162 | except TimeoutError: pass 163 | except websockets.ConnectionClosed: break 164 | except ConnectionResetError: break 165 | except Exception as e:print(f"clients() Ex {e}"); break 166 | return 0 167 | 168 | 169 | 170 | ''' 171 | utility function to parse CLIENTS commands / testing 172 | ''' 173 | async def cmdMmessage(message): 174 | global stop_threads, stop_event 175 | try: 176 | if message == "zzz": ## just testing 177 | print(f"kill sig rec'd"); stop_threads = True 178 | await asyncio.sleep(1); stop_event.set() 179 | elif message == "close": 180 | print(f"recd shutdown from client") 181 | else: 182 | await broadcast_s( message ) 183 | except: pass 184 | return 0 185 | 186 | 187 | 188 | ''' 189 | utility function to print counts of Role-Senders & Role-Clients 190 | ''' 191 | async def stats(): 192 | global ctr 193 | print(f"Clients={ctr[0]}, Senders={ctr[1]}") 194 | 195 | 196 | 197 | ''' 198 | Websocket Server init. 199 | ''' 200 | async def ws_start( stop): 201 | global stop_event, wsport, retCode 202 | print(f"RTD Relay server started: {datetime.datetime.now()} on localhost:{wsport},\nctrl+c once to shutdown.") 203 | try: 204 | async with websockets.serve(handler, "localhost", wsport): 205 | await stop 206 | 207 | except Exception as e: print(f"ws_start() Ex {e}"); stop_event.set(); retCode = 404; return 208 | 209 | 210 | ''' 211 | Main function to create Asyncio Event_Loop and start Websocket Server. 212 | ''' 213 | def main_S( lst=[] ): 214 | global stop_event, stop_threads, wsport, retCode 215 | 216 | try: 217 | if len( lst ) > 2: ## sys.argv[0] = dummy 218 | p = int( lst[1] ) ## try to int 219 | if lst[2] == "startserver": 220 | wsport = p 221 | else: print(f"Bad args"); exit(404) 222 | 223 | except: print(f"first arg is port#"); exit(404) 224 | 225 | try: 226 | loop = asyncio.new_event_loop() 227 | asyncio.set_event_loop( loop) 228 | stop_event = threading.Event() 229 | stop = loop.run_in_executor(None, stop_event.wait) 230 | 231 | loop.run_until_complete( ws_start( stop)) 232 | stop_threads = True; loop.stop(); loop.close() 233 | 234 | except Exception as e: print(f"main() Ex {e}"); stop_threads = True; stop_event.set(); loop.stop() 235 | except BaseException as b:print(f"main() bEx {b}"); stop_threads = True; stop_event.set(); loop.stop() 236 | print(f"shutdown server. Exit {retCode}"); exit( retCode) 237 | 238 | 239 | 240 | if __name__ == "__main__": 241 | if( len( sys.argv) > 2): main_S( sys.argv ) ## default port 10101 242 | else: main_S( ['main', str(wsport), 'startserver' ] ) 243 | 244 | 245 | ## Credits: 246 | ## NSM51, Author. -------------------------------------------------------------------------------- /Samples/Client/Python/client.py: -------------------------------------------------------------------------------- 1 | ''' 2 | // ##### Sample_Client.py ##### 3 | // 4 | // Independent Client-app emulator for WsRtd AmiBroker Data Plugin ( TESTING / DEMO ) 5 | // 6 | // Python Program that runs as a Fake data generator. 7 | // WsRTD data plugin connects to the relay server via specified IP:Port, and this client 8 | // also connects to the relay server. 9 | // 10 | // This program is a template for PRODUCTION USE. 11 | // 12 | /////////////////////////////////////////////////////////////////////// 13 | // Author: NSM51 14 | // https://github.com/ideepcoder/Rtd_Ws_AB_plugin/ 15 | // https://forum.amibroker.com/u/nsm51/summary 16 | // 17 | // Users and possessors of this source code are hereby granted a nonexclusive, 18 | // royalty-free copyright license to use this code in individual and commercial software. 19 | // 20 | // AUTHOR ( NSM51 ) MAKES NO REPRESENTATION ABOUT THE SUITABILITY OF THIS SOURCE CODE FOR ANY PURPOSE. 21 | // IT IS PROVIDED "AS IS" WITHOUT EXPRESS OR IMPLIED WARRANTY OF ANY KIND. 22 | // AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOURCE CODE, 23 | // INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. 24 | // IN NO EVENT SHALL AUTHOR BE LIABLE FOR ANY SPECIAL, INDIRECT, INCIDENTAL, OR 25 | // CONSEQUENTIAL DAMAGES, OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, 26 | // WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, 27 | // ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOURCE CODE. 28 | // 29 | // Any use of this source code must include the above notice, 30 | // in the user documentation and internal comments to the code. 31 | ''' 32 | 33 | ## Make sure to implement both recv and send, dummy if reqd. This program only sent, and was failing with keepalive timeout. 34 | ## its meant to send RTD data to Relay server. 35 | 36 | import asyncio 37 | import websockets 38 | import datetime 39 | import json 40 | import random 41 | import copy 42 | import pandas as pd 43 | from queue import Queue, Empty, Full 44 | 45 | tf = 1 ## base time interval in min (periodicity) 46 | wsport = 10101 ## WS port 10101 8765 47 | sleepT = 0.9 ## simulate tics in seconds 48 | tCount = 5 ## incl default 3 tickers, increase maxTicks accordingly 49 | incSym = 0 ## set greater than 0, to simulate new quotes by this amount. IF 0, no new tickers added 50 | maXTicks = 50 ## maximum concurrent ticks Try 1000 :) no problem 51 | 52 | ''' Globals ''' 53 | stop_flag = False 54 | addrem_list = [] ## simulate add and remove symbol command 55 | sock_send_Q = Queue() ## Queue used for websocket send. Assumes only 1 Remote source. Else move to Handler 56 | sock_recv_Q = Queue() ## incomging Queue buffer. Assumes only 1 Remote source. Else move to Handler 57 | 58 | def r(l=1,u=9): return random.randint(l, u) 59 | 60 | # with the server 61 | async def ws_client(): 62 | global stop_flag, wsport 63 | print(f"WebSocket: Client Connected. {datetime.datetime.now()}") 64 | url = "ws://127.0.0.1:"+ str(wsport) 65 | # Connect to the server 66 | 67 | async with websockets.connect(url ) as websocket: 68 | global stop_flag 69 | global sleepT, tf, tCount, incSym, maXTicks 70 | 71 | s1=s2=s3=0 72 | v1=v2=v3=0 73 | pTm = 0 74 | 75 | try: await websocket.send( "rolesend" ) 76 | except: raise websockets.ConnectionClosed( None, None ) 77 | 78 | asyncio.create_task( process_recv_Q() ) 79 | 80 | asyncio.create_task( recv( websocket) ) 81 | 82 | ## IF TRUE, This part is for testing commands and the RTD GEnerator 83 | ## Testing commands tcmd1 or tcmd2, IF either True, Scripts exits after sending commmand 84 | 85 | tcmd1 = False; sendstr = "" 86 | if tcmd1: 87 | jo = {"cmd":"bfsym","arg":"y ABC1 1"} 88 | sendstr = some_historical_data( jo ) 89 | 90 | tcmd2 = False 91 | if tcmd2: 92 | tmp_dt = 20241124; tmp_tm = 101000 93 | #jo = {"hist":"SYM9","format":"dtohlcvi","bars":[[tmp_dt,tmp_tm,22.1,22.5,22.1,22.8,120,0]]} 94 | #jo = {"cmd":"bfall","code":200,"arg":"ok","file":"G:\\file.txt","format":"G:\\file.format"} 95 | jo = {"cmd":"addsym","arg":"ABC4"} ## Subscribe symbol 96 | #jo = {"cmd":"remsym","arg":"ABC4"} ## Subscribe symbol 97 | jo = {"cmd":"dbremsym","code":300,"arg":"ZZZ1"} ## RemoveSymbolInAb 98 | #jo = {"cmd":"dbgetlist","code":300,"arg":""} 99 | sendstr = json.dumps(jo) 100 | 101 | if tcmd1 or tcmd2: 102 | await websocket.send( sendstr ) 103 | await asyncio.sleep( 5 ) ## await any plug-in responses, important 104 | stop_flag = True 105 | 106 | ## END testing commands 107 | 108 | ## Put code for BROKER / VENDOR Library in this while loop, 109 | ## or create task and await here. It varies depending on their Library 110 | 111 | while not stop_flag: 112 | try: 113 | dt = datetime.datetime.now() 114 | 115 | t = dt.hour*10000 + int( dt.minute/tf )*tf*100 116 | d = int( dt.strftime('%Y%m%d') ) 117 | 118 | if( pTm != t): 119 | v1 =r(3,5); v2 =r(2,3); v3 =r(1,2); pTm = t; # bar vol reset 120 | if( incSym and tCount <= maXTicks): 121 | tCount += 1; print(tCount) 122 | 123 | else: v1+=r(3,5); v2+=r(2,3); v3+=r(1,2) # bar vol cum 124 | 125 | s1+=v1; s2+=v2; s3+=v3 #total vol 126 | 127 | ## Open intentionally kept random, SYM2 test bad symbol 128 | 129 | ##'n', 'd', 't', 'o', 'h', 'l', 'c', 'v', 'oi', 's','pc','bs','bp','as','ap' (s=total vol, pc=prev day close bs,bp,as,ap=bid ask ) 130 | data = [{"n": "SYM1", "t":t, "d":d, "c": r(1,9), "o": r(1,9), "h": 9, "l": 1, "v": v1, "oi": 0, "bp": r(1,5), "ap": r(5,9), "s": s1,"bs":1,"as":1,"pc":1,"do":4,"dh":9,"dl":1} 131 | ,{"n": "", "t":t, "d":d, "c": r(10,19), "o": r(10,19), "h": 19, "l": 10, "v": v2, "oi": 0, "bp": r(10,15), "ap": r(15,19), "s": s2,"pc":10,"do":15,"dh":19,"dl":10} 132 | ,{"n": "SYM3", "t":t, "d":d, "c": r(20,29), "o": r(20,29), "h": 29, "l": 20, "v": v3, "oi": 0, "bp": r(20,25), "ap": r(25,29), "s": s3,"pc":22,"do":28,"dh":29,"dl":20}] 133 | 134 | k = 4 135 | while k <= min( tCount, maXTicks): 136 | rec = {"n": "SYM"+str(k), "t":t, "d":d, "c": 18+r(), "o": 20+r(), "h": 40-r(), "l": 10+r(), "v": v1, "oi": 0, "bp": r(1,5), "ap": r(5,9), "s": s1,"bs":1,"as":1,"pc":1,"do":20,"dh":40,"dl":10} 137 | data.append( rec ) 138 | k +=1 139 | 140 | ## make ticks for subscribed symbols 141 | for asym in addrem_list: 142 | rec = {"n": asym, "t":t, "d":d, "c": 18+r(), "o": 20+r(), "h": 40-r(), "l": 10+r(), "v": v1, "oi": 0, "bp": r(1,5), "ap": r(5,9), "s": s1,"bs":1,"as":1,"pc":1,"do":20,"dh":40,"dl":10} 143 | data.append( rec ) 144 | 145 | 146 | ##print( json.dumps( data, separators=(',', ':')) ) 147 | await websocket.send( json.dumps( data, separators=(',', ':'))) 148 | 149 | try: 150 | rs = sock_send_Q.get( timeout=0.01 ); ## timeout quickly, don't block if empty 151 | await websocket.send( rs ) ## send other messages like backfill/response cmd 152 | except Empty: pass 153 | 154 | 155 | # client disconnected? 156 | except KeyboardInterrupt: stop_flag = True 157 | except websockets.ConnectionClosed as c: print(f"Conn closed {datetime.datetime.now()} {repr(c)}"); break 158 | except Exception as e: print(f"ws() Ex: {e}"); break 159 | except BaseException as b: print(f"ws() bEx: {b}"); break 160 | 161 | await asyncio.sleep( sleepT ) 162 | return 163 | 164 | 165 | ## Proper way is for handler to have 2 task threads for send() and recv(), 166 | # in this case Handler thread work for sending. 167 | # Use a Queue for generator data to push, and send to pop, 168 | # and another for recv(). Then it is truly async 169 | 170 | ## Recv() is non-blocking, requests are pushed into Queue 171 | 172 | async def recv( websocket): 173 | global stop_flag, sock_recv_Q 174 | try: 175 | while not stop_flag: 176 | try: 177 | async with asyncio.timeout(delay=0.5): 178 | mr = await websocket.recv() 179 | try: 180 | sock_recv_Q.put_nowait( mr ) 181 | except Full: pass 182 | 183 | # client disconnected? 184 | except asyncio.TimeoutError: pass 185 | except websockets.ConnectionClosed as e: print(f"recv() {e}") ; return 186 | except Exception as e: print(f"recv() Ex: {e}"); return 187 | except BaseException as b: print(f"recv() bEx: {b}"); return 188 | 189 | 190 | ''' 191 | Implement Broker/Vendor Backfill process here 192 | ''' 193 | def some_historical_data( jo ): 194 | '''simulate some historical data''' 195 | 196 | ## 10:unix timestamp, // 11:unix millisec timestamp, 197 | ## 20:"20171215 091500", // 21:"2017-12-15 09:15:00", // 22:"2017-12-15","09:15:00", 198 | ## 30:20171215,91500, // 31: 20171215,0,(EoD) 199 | DtFormat = 30 ## {unix: 10, 11,} {str:20, 21, 22,} {int: 30(default), } 200 | 201 | try: 202 | t:str = jo['arg'] 203 | t = t.split(' ') ## reserved1 symbol_name date_from date_to timeframe ## to-do: make a json standard format 204 | 205 | ## SAMPLE using DF, how to mainpulate type etc 206 | hd_str = '[["2024-06-11","10:00:00",22.1,22.5,22.1,22.8,12,0],["2024-06-12","10:05:00",22.8,22.7,22.3,22.4,28,0],\ 207 | ["2024-06-13","10:10:00",22.8,22.7,22.3,22.4,28,0],["2024-06-14","10:15:00",22.8,22.7,22.3,22.4,28,0],\ 208 | ["2024-06-15","10:20:00",22.8,22.7,22.3,22.4,28,0]]' 209 | 210 | df = pd.DataFrame( json.loads( hd_str) ) ## simulate DF 211 | #df['oi'] = 0 212 | 213 | if DtFormat in [30, 31, 32]: ## sample conversion 214 | df.columns = df.columns.astype(str) ## change type of columns from int to str 215 | 216 | df['0'] = df['0'].str.replace('-','') 217 | df['1'] = df['1'].str.replace(':','') 218 | df = df.astype( {"0": "int32", "1": "int32"} ) 219 | 220 | 221 | ## fail safe 222 | if len(t) != 3: t = ["y", "SYM2", 1] 223 | 224 | ## simulate error resopnse message in backfill 225 | if t[1]=="SYM2": 226 | ## SYM2 is also a bad json RTD (note) 227 | jsWs = {"cmd":"bfsym","code":404,"arg":"example of backfill error in "+t[1]} 228 | return json.dumps( jsWs, separators=(',', ':') ) 229 | 230 | ## simulate ASCII import response message 231 | elif t[1]=="SYM6": 232 | jsWs = {"cmd":"bfsym","code":200,"arg":"example ASCII import"+t[1],"file":"D:\\test_ascii_import.txt","format":"D:\\test_ascii.format"} 233 | return json.dumps( jsWs, separators=(',', ':') ) 234 | 235 | #### This is backfill data generator, DF above just for illustration #### 236 | 237 | jsWs = {'hist':t[1], 'format':"dtohlcvi"} ## unix: "uohlcvi", you can re-arrange the fields 238 | 239 | ## Sample using DF. make sure columns match the format 240 | #jsWs['bars'] = 0 241 | ## join to form json string with df output 242 | '''jsStr = json.dumps( jsWs, separators=(',', ':') ) ## , default=int argument 243 | jsStr = jsStr[:-2] 244 | jsStr = jsStr + df.to_json( orient="values" ) 245 | jsStr = jsStr + '}' 246 | return jsStr''' 247 | 248 | ### Below is a dummy historical data generator ### 249 | jsWs['bars'] = [] 250 | 251 | bfdays = int( t[2] ) # days argument 252 | BarsPerDay = 3 # no of bars to generate 253 | tf = 1 # timeframe in minutes 254 | dt = datetime.datetime.now() 255 | dt = dt - datetime.timedelta( days=bfdays, minutes=( (bfdays+1)*BarsPerDay ) ) 256 | 257 | i = 0 258 | while i <= bfdays: 259 | j = 0 260 | while j < BarsPerDay: 261 | 262 | jsWs['bars'].append( [ int( dt.strftime('%Y%m%d') ), int( dt.strftime('%H%M00') ), 263 | 20+r(),40-r(),10+r(),18+r(),100+r(100,500),0] ) 264 | 265 | ## Unix time example, change format string above [use u = localtime() or g=gmtime() c++] 266 | #jsWs['bars'].append( [ int( dt.timestamp()), 20+r(),40-r(),10+r(),18+r(),100+r(100,500),0] ) 267 | 268 | dt = dt + datetime.timedelta( minutes=tf ) 269 | j +=1 270 | 271 | dt = dt + datetime.timedelta( days=1 ) 272 | i += 1 273 | 274 | return json.dumps( jsWs, separators=(',', ':') ) ## remove space else plugin will not match str 275 | 276 | except Exception as e: 277 | return repr(e) 278 | 279 | 280 | ## simulate subscribe new symbol in Remote Server 281 | def add_symbol( jo ): 282 | global addrem_list 283 | 284 | jr = copy.deepcopy( jo ) 285 | sym = jr['arg'] 286 | 287 | if sym not in addrem_list: 288 | addrem_list.append( sym ) 289 | jr['code'] = 200 290 | jr['arg'] = sym + " subscribed ok" 291 | else: 292 | jr['code'] = 400 293 | jr['arg'] = sym + " already subcribed" 294 | 295 | return json.dumps( jr, separators=(',', ':') ) 296 | 297 | 298 | ## simulate unsubscribe symbool from remote server 299 | def rem_symbol( jo ): 300 | global addrem_list 301 | 302 | jr = copy.deepcopy( jo ) 303 | sym = jr['arg'] 304 | 305 | if sym not in addrem_list: 306 | jr['code'] = 400 307 | jr['arg'] = sym + " not subscribed" 308 | else: 309 | addrem_list.remove( sym ) 310 | jr['code'] = 200 311 | jr['arg'] = sym + " unsubcribed ok" 312 | 313 | return json.dumps( jr, separators=(',', ':') ) 314 | 315 | 316 | ''' 317 | Receive() Websocket is asynchronous. This means requests do NOT block the websocket. 318 | Messages are pushed into sock_recv_Q Queue, and this async task will process them at 319 | its own pace. This will prevent RATE LIMITS being hit by Broker/Vendor. 320 | ''' 321 | async def process_recv_Q(): 322 | global stop_flag, sock_recv_Q 323 | 324 | while not stop_flag: 325 | try: 326 | mr = sock_recv_Q.get_nowait() 327 | except Empty: 328 | await asyncio.sleep( 0.2 ) ## prevent hot loop 329 | continue 330 | 331 | print(mr) 332 | if mr == "zzc": 333 | print(f"kill sig rec'd") 334 | stop_flag = True # stop all loops 335 | await asyncio.sleep(1) # wait task complete 336 | 337 | try: 338 | jo = json.loads( mr ) 339 | if 'cmd' in jo: 340 | if 'arg' in jo: 341 | if jo['cmd']=='bfall': 342 | print( f"bfall cmd in {mr}") 343 | elif jo['cmd'] in ['bfauto', "bffull"]: 344 | print( f"bfauto/bffull cmd in {mr}") 345 | 346 | sym = jo['arg'] if ' ' not in jo['arg'] else (jo['arg'].split(' '))[0] 347 | jo['arg'] = f"y {sym} 5" if jo['cmd']=='bfull' else f"y {sym} 2" 348 | 349 | try: 350 | sock_send_Q.put( some_historical_data( jo ) ) #; print( ret ) 351 | except Full: print(f"recv() Ex: send.Q full"); pass 352 | 353 | jo['sym'] = "addsym" 354 | jo['arg'] = sym 355 | add_symbol( jo ) 356 | 357 | elif jo['cmd'] == 'bfsym': 358 | #hist_Q.put( jo ) ## real code should use Queue as buffer, separate thread/async 359 | ## jo = {"cmd":"bfsym", "arg":"y SYM1 3 1"} 360 | try: 361 | sock_send_Q.put( some_historical_data( jo ) ) #; print( ret ) 362 | except Full: print(f"recv() Ex: send.Q full"); pass 363 | print( f"sent response\n{jo}" ) 364 | 365 | elif jo['cmd'] == "addsym": 366 | try: 367 | jr = add_symbol( jo ) 368 | sock_send_Q.put( jr ) #; print( ret ) 369 | except Full: print(f"recv() Ex: send.Q full"); pass 370 | print( f"sent response\n{jr}" ) 371 | 372 | elif jo['cmd'] == "remsym": 373 | try: 374 | jr = rem_symbol( jo ) 375 | sock_send_Q.put( jr ) #; print( ret ) 376 | except Full: print(f"recv() Ex: send.Q full"); pass 377 | print( f"sent response\n{jr}" ) 378 | 379 | else: print( f"unknown cmd in {mr}") 380 | 381 | else: print( f"arg not found in {mr}") 382 | 383 | else: print( f"jo={mr}") 384 | 385 | except ValueError as e: 386 | #print(e) ## if not JSON 387 | print( mr ) 388 | 389 | 390 | 391 | 392 | def main(): 393 | global stop_flag 394 | try: 395 | # Start the connection 396 | asyncio.run( ws_client() ) 397 | 398 | except KeyboardInterrupt: print(f"KB exit"); stop_flag = True 399 | except Exception as e: print(f"main() Ex {e}"); stop_flag = True 400 | except BaseException as b: print(f"main() bEx {b}"); stop_flag = True 401 | 402 | print(f"client exited ok.") 403 | exit() 404 | 405 | 406 | if __name__ == '__main__': 407 | main() 408 | 409 | ## For more comments or Documentation, see the Sample_Server.py 410 | ## This has the same functions, only difference is that this is 411 | ## a client websocket connection to relay server, 412 | ## whereas sample_server bundles both websocket relay server and client generator. -------------------------------------------------------------------------------- /Samples/Client/Python/notice.txt: -------------------------------------------------------------------------------- 1 | This plugin should be distributed or redisributed with the Documentation 2 | file provided in HTML that contains all the Licence information, 3 | Trademark, Copyright & Other notices. -------------------------------------------------------------------------------- /Samples/Client/Python/vendor_class_wrapper.py: -------------------------------------------------------------------------------- 1 | ''' 2 | // ##### vendor_class.py ##### 3 | // 4 | // Client-app Class wrapper for Broker/Vendor library for WsRtd AmiBroker Data Plugin 5 | // 6 | // Python Program that can be used to wrap a Broker/Vendor Library. 7 | // Key highlights is it has a dataframe to Build Bars for base time interval. 8 | // Data transmission works with snapshotting model 9 | // 10 | // This program is a template for PRODUCTION USE. 11 | // 12 | /////////////////////////////////////////////////////////////////////// 13 | // Author: NSM51 14 | // https://github.com/ideepcoder/Rtd_Ws_AB_plugin/ 15 | // https://forum.amibroker.com/u/nsm51/summary 16 | // 17 | // Users and possessors of this source code are hereby granted a nonexclusive, 18 | // royalty-free copyright license to use this code in individual and commercial software. 19 | // 20 | // AUTHOR ( NSM51 ) MAKES NO REPRESENTATION ABOUT THE SUITABILITY OF THIS SOURCE CODE FOR ANY PURPOSE. 21 | // IT IS PROVIDED "AS IS" WITHOUT EXPRESS OR IMPLIED WARRANTY OF ANY KIND. 22 | // AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOURCE CODE, 23 | // INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. 24 | // IN NO EVENT SHALL AUTHOR BE LIABLE FOR ANY SPECIAL, INDIRECT, INCIDENTAL, OR 25 | // CONSEQUENTIAL DAMAGES, OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, 26 | // WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, 27 | // ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOURCE CODE. 28 | // 29 | // Any use of this source code must include the above notice, 30 | // in the user documentation and internal comments to the code. 31 | ''' 32 | 33 | 34 | import time 35 | import datetime 36 | import pandas as pd 37 | 38 | from Real_Vendor_Class import DataSocket ## Your actual Vendor Class 39 | ## OR 40 | import DataSocket 41 | 42 | 43 | ## Global variables 44 | cdt = 0 ## current datetime.datetime.now() object 45 | V_DS_ctr = 0 ## Vendor connection error counter 46 | secTick = 1 ## one minute bars, +ve int for minute. (1,5,15 AB supported) Later, 0=1 sec 47 | 48 | 49 | ## Global objects 50 | vendor_ws_1 = 0 ## our class data socket instance 51 | printDFlist = [] ## Storage List to print symbols. Debugging. 52 | 53 | 54 | ############ Functions ############ 55 | 56 | def GetAuthToken(): 57 | '''use vendor code to generate API Key or Authentication Token''' 58 | 59 | return "" 60 | 61 | 62 | ## remember to keep updating cdt 63 | def rcdt()->datetime.datetime: 64 | '''Returns as well as Updates global current-datetime variable cdt.''' 65 | 66 | global cdt; cdt = datetime.datetime.now(); return cdt 67 | rcdt() 68 | 69 | 70 | def Scdt(i=0): 71 | '''DT as string''' 72 | global cdt 73 | return cdt.strftime('%H:%M') if i else cdt.strftime('%H:%M:%S') 74 | 75 | 76 | ## Make Todays DateTime from TIME arguments 77 | def makeTime( H:int=0, M:int=0, S:int=0 )->datetime.datetime: 78 | cdt = datetime.datetime.now() 79 | 80 | if isinstance( H, str) and M==0 and S==0: 81 | sTime = H.split(sep=':') 82 | H=int( sTime[0] ); M=int( sTime[1]); S=int( sTime[2] ) 83 | return datetime.datetime( cdt.year,cdt.month,cdt.day,H,M,S ) 84 | 85 | 86 | 87 | def barSec(t:datetime.datetime): return t.hour*10000 + t.minute*100 + t.second # second ticks 88 | 89 | def barmin(t:datetime.datetime): return t.hour*10000 + t.minute//secTick*100*secTick # create candle time in min 90 | 91 | candle = barmin if secTick else barSec ## point to candle func, for base time interval 92 | 93 | 94 | 95 | def Rename_Symbol( s ): 96 | '''This is where you can perform symbol renaming''' 97 | 98 | return "" 99 | 100 | 101 | class VendorDataSocket(): 102 | 103 | def __init__(self, symbols:list, reconnect=True, rtd_type="RtdTick"): 104 | 105 | super().__init__() 106 | 107 | ## Some common parameters used by Vendor 108 | self.access_token = GetAuthToken() 109 | self.reconnect = reconnect 110 | self.symbols = list( filter(None, symbols)) ## List of subscribed symbols 111 | self.data_type = rtd_type 112 | 113 | ## Pointer for Vendor Class 114 | self.vendor = None 115 | 116 | ## Some debugging functions to expose 117 | self.printMsgsF = False 118 | self.debugSym = 0 119 | 120 | 121 | ## WsRtd DF 122 | ##pv=prev bar vol, wl=snapshot price, wv=snapshot vol, s=Total_daily_vol, symbol=feed_sym_name,n=sym_renamed_if_reqd 123 | 124 | self.goodtick = int( makeTime( 9,15,0 ).timestamp()) ## timestamp filter, say skip pre-open market 125 | self.clsNm = 'RTD:' 126 | self.dfcols = ['symbol', 'n', 'd', 't', 'o', 'h', 'l', 'c', 'v', 'oi', 'pv', 'wl', 'wv', 's','pc','bs','bp','as','ap','do','dh','dl','ch','cp'] 127 | self.outcols = [ 'n', 'd', 't', 'o', 'h', 'l', 'c', 'v', 'oi', 's','pc','bs','bp','as','ap','do','dh','dl','ch','cp'] 128 | self.ed = {'d':int(cdt.strftime('%Y%m%d'))} # rest init to 0 129 | 130 | ## init code for DataFrame 131 | self.df = pd.DataFrame( columns=self.dfcols) 132 | self.df = self.df.astype({'symbol':str, 'n':str, 'd':int, 't':int, 'o':float, 'h':float, 'l':float, 'c':float, 'v':int, 133 | 'oi':int, 'pv':int, 'wl':float, 'wv':int, 's':int, 'pc':float, 'bs':int,'bp':float,'as':int,'ap':float, 134 | 'do':float, 'dh':float, 'dl':float, 'ch':float,'cp':float }) 135 | 136 | self.df.set_index('symbol', inplace=True ) 137 | 138 | 139 | def on_error(self, message): 140 | global V_DS_ctr 141 | V_DS_ctr += 1 142 | s = f"{self.clsNm} error: {Scdt()} {message}" 143 | print( s ) 144 | if V_DS_ctr > 10: print("code to take action as error counter hit") 145 | 146 | 147 | def on_disconnect(self, msg): 148 | try: 149 | rcdt() 150 | if msg['code']==200: print(f"{self.clsNm} close: {Scdt()} {msg['message']}") 151 | except: print(f"{self.clsNm} close: {Scdt()} {msg}") 152 | 153 | '''cleanup and post disconnect here''' 154 | 155 | 156 | def on_connect(self): 157 | self.subscribe_symbol( self.symbols) 158 | self.vendor.run_thread() ## vendor defined function to keep it running 159 | 160 | 161 | def connect(self): 162 | 163 | ## Assign CallBack functions here for Vendor Library and start. Just a sample with expected callbacks 164 | 165 | self.vendor = Real_Vendor_Class.DataSocket( 166 | token = self.access_token, 167 | reconnect = self.reconnect, 168 | on_connect = self.on_connect, 169 | on_disconnect = self.on_disconnect, 170 | on_error = self.on_error, 171 | on_data = self.on_data, 172 | retry_connect = 10 173 | ) 174 | self.vendor.connect() ## this is where you start you vendor's thread/class 175 | 176 | 177 | def addSym(self, s): 178 | if s not in self.symbols: self.symbols.append( s ) ##List not really used, just keeps List of symbols and init variable 179 | 180 | if s not in self.df.index: 181 | self.df.loc[s] = 0 182 | self.df.at[s, 'n'] = Rename_Symbol( s ) 183 | self.df.loc[s, self.ed.keys()] = self.ed.values() ## easy way to assign dict 184 | 185 | 186 | def subscribe_symbol(self, sym:list): 187 | '''takes LIST of symbols to Subscribe''' 188 | sym = list( filter( None, sym)) # remove '' in list 189 | if sym == []: return 190 | 191 | for s in sym: self.addSym(s) 192 | 193 | self.symbols.sort() 194 | self.df.sort_index( inplace=True ) 195 | self.vendor.subscribe( symbols=sym ) #, channel=15 196 | print(f"{self.clsNm} subs: { ', '.join(sym) }") 197 | 198 | 199 | def unsubscribe_symbol(self, sym:list[str] ): 200 | '''takes LIST of symbols to Unsubscribe''' 201 | sym = list( filter( None, sym)) # remove '' in list 202 | if sym == []: return 203 | 204 | self.vendor.unsubscribe( symbols=sym ) # vendor class to unsubscribe 205 | time.sleep(0.5) # wait for round trip 206 | ## unsub first, then cleanup 207 | for s in sym: 208 | if s in self.symbols: 209 | self.symbols.remove( s ) 210 | if s in self.df.index: 211 | self.df.drop(s, inplace=True) 212 | 213 | self.symbols.sort() 214 | self.df.sort_index( inplace=True ) 215 | print(f"{self.clsNm} unsub: { ', '.join(sym) }") 216 | 217 | 218 | def disconnect(self): 219 | try: 220 | if self.vendor: self.vendor.close_connection() 221 | except Exception as e: 222 | print(f'{self.clsNm} disconnect(): {repr(e)}') 223 | 224 | 225 | def on_data(self, msg): 226 | '''most important function that processes messages from vendor''' 227 | try: 228 | #t1 = time.perf_counter() 229 | if "data" in msg: 230 | if msg["data"] in ('stock', 'index'): ## Example: IF message type is stock or index 231 | 232 | s=msg['symbol']; sf=0; tf='exch_time' ## tf for Index/non-index symbols 233 | 234 | ## testing, output msg to stdio 235 | if self.printMsgsF or self.debugSym==s: print( msg ) 236 | 237 | #Filter some pre=open ticks, for example 238 | if self.goodtick > msg[tf]: return 239 | 240 | if msg["data"]=='stock': 241 | pvol=self.df.at[s, 'pv']; tvol=msg['total_volume'] 242 | if not (pvol+self.df.at[s, 'v']) < tvol: return ## update flag=0, writeDF skips it. RETURN as NO Vol+ 243 | else: sf=1; tf='last_trade_time' 244 | 245 | p = msg['last_trade_price'] # result=df.loc[s]## extract row, but loses index:not reqd 246 | t = candle( datetime.datetime.fromtimestamp( msg[tf])) # last time, create bar time (barSec()/barMin()) 247 | 248 | 249 | ## Now, we Build bars of base time interval 250 | 251 | if t == self.df.at[s, 't']: 252 | if p > self.df.at[s, 'h']: self.df.at[s, 'h'] = p 253 | if p < self.df.at[s, 'l']: self.df.at[s, 'l'] = p 254 | if sf: self.df.at[s, 'v']= tvol - pvol 255 | else: 256 | self.df.at[s, 'o']= self.df.at[s, 'h']= self.df.at[s, 'l']= p ##OHL=p 257 | if sf: 258 | self.df.at[s, 'v'] = msg['last_traded_qty'] 259 | self.df.at[s, 'pv'] = tvol - msg['last_traded_qty'] 260 | 261 | self.df.at[s, 'c'] = p 262 | self.df.at[s, 't'] = t 263 | self.df.at[s, 'pc'] = msg["prev_day_close"] 264 | self.df.at[s, 'do'] = msg["open_price"] ## For RT Window 265 | self.df.at[s, 'dh'] = msg["high_price"]; self.df.at[s, 'dl'] = msg["low_price"] ## Day Open, Day High, Day Low 266 | self.df.at[s, 'ch'] = msg["ch"]; self.df.at[s, 'cp'] = msg["chp"] ## perc_change and change 267 | 268 | ## Make bars for indices or tradable instruments 269 | if sf: 270 | self.df.at[s, 's'] = tvol 271 | self.df.at[s, 'bs'] = msg['bid_size']; self.df.at[s,'bp']=msg['bid_price']; self.df.at[s,'as']=msg['ask_size']; self.df.at[s,'ap']=msg['ask_price'] 272 | else: 273 | self.df.at[s, 's'] = 0 274 | self.df.at[s, 'bs'] = 0; self.df.at[s,'bp']=0; self.df.at[s,'as']=0; self.df.at[s,'ap']=0 ## indices are 0 275 | 276 | elif msg["data"] in ('conn','other','subscribe','unsubscribe'): 277 | if msg['code']==200: print(f"{self.clsNm} msg: {msg['message']}") 278 | else: print(f"{self.clsNm} msg: code={msg['code']} {msg['message']}") 279 | 280 | else: print( msg ) 281 | 282 | else: print( f'{self.clsNm} msg_unkown:', msg ) 283 | #t2=time.perf_counter(); print(f'on_msg: {t2-t1}') ##2500 call/s or max 0.0004 per call 284 | 285 | except Exception as e: 286 | 287 | if msg['symbol'] not in self.df.index: 288 | print(f'{self.clsNm} msg ex: {msg['symbol']} not in DF') 289 | self.addSym( msg['symbol']) 290 | 291 | else: print(f'{self.clsNm} msg ex: {msg['symbol']} {e}') 292 | 293 | return 0 294 | 295 | 296 | def printDF(self): 297 | '''Debug and print few fields of a specific symbol''' 298 | 299 | global cdt, printDFlist 300 | try: 301 | if len(printDFlist) < 1: return 0 302 | 303 | d = self.df.query(f't == {candle(cdt)} and c != wl and (v != wv or v == 0) and n in {printDFlist}')[['n','t','c','v']] 304 | 305 | if d.empty: return 0 306 | else: print(d); return 1 307 | 308 | except Exception as e: 309 | print(f'{self.clsNm} print(): {e}') 310 | return 0 311 | 312 | 313 | def writeDF(self): 314 | '''Snapshot the DF here and send RTD to socket''' 315 | 316 | rcdt() 317 | #;t1=time.perf_counter() 318 | 319 | global cdt #, outData 320 | try: 321 | # {t}and c != wl and (v != wv or v == 0) # vol_chg OR ltp_chg, less strict 322 | d = self.df.query(f't == {candle(cdt)} and (v != wv or (c != wl and v == 0))') 323 | 324 | if d.empty: return 0 325 | 326 | j = d[self.outcols].to_json( orient="records" ) 327 | 328 | ## Log messages to file 329 | #d.to_csv( outData, columns=['n', 'd', 't', 'o', 'h', 'l', 'c', 'v', 'oi'], index=False, header=False, mode='w') 330 | 331 | self.df['wl'] = self.df['c']; self.df['wv'] = self.df['v'] ##ok on FULL df, use assign() and operating on slice throws issues. 332 | 333 | #t2=time.perf_counter(); print(f'writeDF: {t2-t1}') ## performance timer 334 | 335 | return j ## JSON in WS mode, #return 1 ## in AB.Import mode 336 | except Exception as e: 337 | print(f'{self.clsNm} writeDF(): {e}') 338 | return 0 339 | 340 | ### Below 3 Debugging functions ### 341 | 342 | def printCurrList(self): print(f"{self.clsNm} current list: {', '.join( self.symbols ) }"); return 0 343 | 344 | 345 | def funcPrintMsg(self): 346 | if self.printMsgsF: self.printMsgsF = False 347 | else: self.printMsgsF = True 348 | return 0 ## Debug func to set Flag to print WS msgs to stdio. 349 | 350 | 351 | def funcDebugSym(self, a): 352 | if a in self.symbols: self.debugSym = a 353 | else: self.debugSym = None 354 | 355 | 356 | 357 | ####### USAGE ###### 358 | 359 | ## some symbols 360 | symbs = [ 'SYM1', 'SYM2', 'SYM3', 'SYM4', 'SYM5' ] 361 | 362 | ## instantiate data class 363 | vendor_ws_1 = VendorDataSocket( symbols=symbs ) 364 | 365 | ## start data socket 366 | vendor_ws_1.connect() 367 | 368 | 369 | ### other usages ### 370 | 371 | ## loop this to push data, like in client.py 372 | vendor_ws_1.writeDF() 373 | 374 | ## Unsubscribe and Subscribe 375 | vendor_ws_1.unsubscribe_symbol( [ 'SYM2', 'SYM4'] ) 376 | 377 | vendor_ws_1.subscribe_symbol( [ 'SYM6', 'SYM7' ] ) 378 | 379 | 380 | ## 3 various debug functions 381 | vendor_ws_1.printDF() 382 | 383 | 384 | ## disconnect socket 385 | vendor_ws_1.disconnect() -------------------------------------------------------------------------------- /Samples/Server/Python/notice.txt: -------------------------------------------------------------------------------- 1 | This plugin should be distributed or redisributed with the Documentation 2 | file provided in HTML that contains all the Licence information, 3 | Trademark, Copyright & Other notices. -------------------------------------------------------------------------------- /Samples/Server/Python/sample_server.py: -------------------------------------------------------------------------------- 1 | ''' 2 | // ##### Sample_Server.py ##### 3 | // 4 | // Independent RTD generator server for WsRtd AmiBroker Data Plugin ( TESTING / DEMO ) 5 | // 6 | // Python Program that runs as a Fake data generator Server. 7 | // WsRTD data plugin connects to this server via specified IP:Port 8 | // 9 | // This program is NOT meant for PRODUCTION USE. IT is just a tester script. 10 | // 11 | /////////////////////////////////////////////////////////////////////// 12 | // Author: NSM51 13 | // https://github.com/ideepcoder/Rtd_Ws_AB_plugin/ 14 | // https://forum.amibroker.com/u/nsm51/summary 15 | // 16 | // Users and possessors of this source code are hereby granted a nonexclusive, 17 | // royalty-free copyright license to use this code in individual and commercial software. 18 | // 19 | // AUTHOR ( NSM51 ) MAKES NO REPRESENTATION ABOUT THE SUITABILITY OF THIS SOURCE CODE FOR ANY PURPOSE. 20 | // IT IS PROVIDED "AS IS" WITHOUT EXPRESS OR IMPLIED WARRANTY OF ANY KIND. 21 | // AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOURCE CODE, 22 | // INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. 23 | // IN NO EVENT SHALL AUTHOR BE LIABLE FOR ANY SPECIAL, INDIRECT, INCIDENTAL, OR 24 | // CONSEQUENTIAL DAMAGES, OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, 25 | // WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, 26 | // ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOURCE CODE. 27 | // 28 | // Any use of this source code must include the above notice, 29 | // in the user documentation and internal comments to the code. 30 | ''' 31 | 32 | import asyncio 33 | import websockets 34 | import datetime 35 | import json 36 | import random 37 | import sys 38 | import pandas as pd 39 | import copy 40 | from queue import Queue, Empty, Full 41 | 42 | ''' Settings ''' 43 | tf = 1 ## base time interval in min (periodicity) 44 | wsport = 10102 ## Websocket port 10101 45 | sleepT = 0.9 ## simulate ticks generated every "n" seconds. SET IN MAIN() 46 | tCount = 5 ## incl default 3 tickers, increase maxTicks accordingly 47 | incSym = 0 ## set greater than 0, to simulate new quotes by this amount. IF 0, no new tickers added 48 | maXTicks = 50 ## maximum concurrent ticks Try 1000 :) no problem 49 | 50 | KeepRunning = True ## True=Server remains running on client disconnect, False=Server stops on client disconnect 51 | 52 | ''' Globals ''' 53 | addrem_list = [] ## simulate add and remove symbol command 54 | 55 | stop_threads = False ## global flag to send term signal 56 | 57 | 58 | class PubSub: 59 | def __init__(self): 60 | self.waiter = asyncio.Future() 61 | 62 | def publish(self, value): 63 | waiter, self.waiter = self.waiter, asyncio.Future() 64 | waiter.set_result((value, self.waiter)) 65 | 66 | async def subscribe(self): 67 | global stop_threads 68 | 69 | waiter = self.waiter 70 | while not stop_threads: 71 | value, waiter = await waiter 72 | yield value 73 | 74 | def __del__(self): 75 | return 76 | 77 | __aiter__ = subscribe 78 | 79 | 80 | PUBSUB = PubSub() 81 | 82 | 83 | ## Proper way is for handler to have 2 task threads for send() and recv(), 84 | # in this case Handler thread work for sending. 85 | # Use a Queue for generator data to push, and send to pop, 86 | # and another for recv(). Then it is truly async 87 | 88 | async def handler( websocket ): 89 | print(f"client connected") 90 | 91 | global stop_threads 92 | 93 | asyncio.create_task( recv( websocket ) ) 94 | 95 | ## Send() task within handler 96 | try: 97 | while( not stop_threads ): 98 | 99 | async for message in PUBSUB: 100 | await websocket.send( message ) ## send broadcast RTD messages 101 | 102 | if( stop_threads ): 103 | raise websockets.ConnectionClosed( None, None ) 104 | 105 | except websockets.ConnectionClosed as wc: 106 | ## can check reason for close here 107 | if not KeepRunning: stop_threads = True 108 | 109 | except ConnectionResetError: pass 110 | 111 | print(f"client disconnected") 112 | return 113 | 114 | 115 | ## Recv() is blocking while processing 116 | ## in production, push requests to Queue and process asynchronously 117 | ## should not block or use same thread to process requests 118 | 119 | async def recv( websocket ): 120 | 121 | global stop_threads 122 | 123 | try: 124 | while( not stop_threads ): 125 | try: 126 | async with asyncio.timeout(delay=0.3): 127 | mr = await websocket.recv() 128 | try: 129 | jo = json.loads( mr ) 130 | if 'cmd' in jo: 131 | if 'arg' in jo: 132 | if jo['cmd']=='bfall': 133 | print( f"bfall cmd in {mr}") 134 | 135 | elif jo['cmd'] in ['bfauto', 'bffull']: 136 | print( f"bfauto cmd in {mr}") 137 | 138 | sym = jo['arg'] if ' ' not in jo['arg'] else (jo['arg'].split(' '))[0] 139 | 140 | jo['arg'] = f"y {sym} 2" if jo['cmd']=='bfauto' else f"y {sym} 5" 141 | 142 | await broadcast( some_historical_data( jo ) ) 143 | 144 | jo['sym'] = "addsym"; jo['arg'] = sym 145 | add_symbol( jo ) 146 | 147 | elif jo['cmd'] == 'bfsym': 148 | #Sock_send_Q.put( jo ) ## real code should use Queue as buffer, separate thread/async 149 | ## jo = {"cmd":"bfsym", "arg":"y SYM1 3 1"} 150 | await broadcast( some_historical_data( jo ) ) 151 | print( f"sent response\n{jo}" ) 152 | 153 | elif jo['cmd'] == "addsym": 154 | jr = add_symbol( jo ) 155 | await broadcast( jr ) 156 | print( f"sent response\n{jr}" ) 157 | 158 | elif jo['cmd'] == "remsym": 159 | jr = rem_symbol( jo ) 160 | await broadcast( some_historical_data( jr ) ) 161 | print( f"sent response\n{jr}" ) 162 | 163 | else: print( f"unknown cmd in {mr}") 164 | 165 | else: print( f"arg not found in {mr}") 166 | 167 | else: print( f"jo={mr}") 168 | 169 | except ValueError as e: 170 | #print(e) ## if not JSON 171 | print( mr ) 172 | 173 | if stop_threads: 174 | raise websockets.ConnectionClosed( None, None ) 175 | 176 | except TimeoutError: pass 177 | 178 | except websockets.ConnectionClosed as wc: 179 | if not KeepRunning: stop_threads = True 180 | 181 | except Exception as e: 182 | return repr(e) 183 | 184 | return 185 | 186 | 187 | def some_historical_data( jo ): 188 | '''simulate some historical data''' 189 | 190 | ## 10:unix timestamp, // 11:unix millisec timestamp, 191 | ## 20:"20171215 091500", // 21:"2017-12-15 09:15:00", // 22:"2017-12-15","09:15:00", 192 | ## 30:20171215,91500, // 31: 20171215,0,(EoD) 193 | DtFormat = 30 ## {unix: 10, 11,} {str:20, 21, 22,} {int: 30(default), } 194 | 195 | try: 196 | t:str = jo['arg'] 197 | t = t.split(' ') ## reserved1 symbol_name date_from date_to timeframe ## to-do: make a json standard format 198 | 199 | ## SAMPLE using DF, how to mainpulate type etc 200 | hd_str = '[["2024-06-11","10:00:00",22.1,22.5,22.1,22.8,12,0],["2024-06-12","10:05:00",22.8,22.7,22.3,22.4,28,0],\ 201 | ["2024-06-13","10:10:00",22.8,22.7,22.3,22.4,28,0],["2024-06-14","10:15:00",22.8,22.7,22.3,22.4,28,0],\ 202 | ["2024-06-15","10:20:00",22.8,22.7,22.3,22.4,28,0]]' 203 | 204 | df = pd.DataFrame( json.loads( hd_str) ) ## simulate DF 205 | #df['oi'] = 0 206 | 207 | if DtFormat in [30, 31, 32]: ## sample conversion 208 | df.columns = df.columns.astype(str) ## change type of columns from int to str 209 | 210 | df['0'] = df['0'].str.replace('-','') 211 | df['1'] = df['1'].str.replace(':','') 212 | df = df.astype( {"0": "int32", "1": "int32"} ) 213 | 214 | 215 | ## fail safe 216 | if len(t) != 3: t = ["y", "SYM2", 1] 217 | 218 | ## simulate error resopnse message in backfill 219 | if t[1]=="SYM2": 220 | ## SYM2 is also a bad json RTD (note) 221 | jsWs = {"cmd":"bfsym","code":404,"arg":"example of backfill error in "+t[1]} 222 | return json.dumps( jsWs, separators=(',', ':') ) 223 | 224 | ## simulate ASCII import response message 225 | #elif t[1]=="SYM6": 226 | # jsWs = {"cmd":"bfsym","code":200,"arg":"example ASCII import"+t[1],"file":"D:\\test_ascii_import.txt","format":"D:\\test_ascii.format"} 227 | # return json.dumps( jsWs, separators=(',', ':') ) 228 | 229 | #### This is backfill data generator, DF above just for illustration #### 230 | 231 | jsWs = {'hist':t[1], 'format':"dtohlcvi"} ## unix: "uohlcvi", you can re-arrange the fields 232 | 233 | ## Sample using DF. make sure columns match the format 234 | #jsWs['bars'] = 0 235 | ## join to form json string with df output 236 | '''jsStr = json.dumps( jsWs, separators=(',', ':') ) ## , default=int argument 237 | jsStr = jsStr[:-2] 238 | jsStr = jsStr + df.to_json( orient="values" ) 239 | jsStr = jsStr + '}' 240 | return jsStr''' 241 | 242 | ### Below is a dummy historical data generator ### 243 | jsWs['bars'] = [] 244 | 245 | bfdays = int( t[2] ) # days argument 246 | BarsPerDay = 3 # no of bars to generate 247 | tf = 1 # timeframe in minutes 248 | dt = datetime.datetime.now() 249 | dt = dt - datetime.timedelta( days=bfdays, minutes=( (bfdays+1)*BarsPerDay ) ) 250 | 251 | i = 0 252 | while i <= bfdays: 253 | j = 0 254 | while j < BarsPerDay: 255 | 256 | jsWs['bars'].append( [ int( dt.strftime('%Y%m%d') ), int( dt.strftime('%H%M00') ), 257 | 20+r(),40-r(),10+r(),18+r(),100+r(100,500),0] ) 258 | 259 | ## Unix time example, change format string above [use u = localtime() or g=gmtime() c++] 260 | #jsWs['bars'].append( [ int( dt.timestamp()), 20+r(),40-r(),10+r(),18+r(),100+r(100,500),0] ) 261 | 262 | dt = dt + datetime.timedelta( minutes=tf ) 263 | j +=1 264 | 265 | dt = dt + datetime.timedelta( days=1 ) 266 | i += 1 267 | 268 | return json.dumps( jsWs, separators=(',', ':') ) ## remove space else plugin will not match str 269 | 270 | except Exception as e: 271 | return repr(e) 272 | 273 | 274 | ## simulate subscribe 275 | def add_symbol( jo ): 276 | global addrem_list 277 | 278 | jr = copy.deepcopy( jo ) 279 | sym = jr['arg'] 280 | 281 | if sym not in addrem_list: 282 | addrem_list.append( sym ) 283 | jr['code'] = 200 284 | jr['arg'] = sym + " subscribed ok" 285 | else: 286 | jr['code'] = 400 287 | jr['arg'] = sym + " already subcribed" 288 | 289 | return json.dumps( jr, separators=(',', ':') ) 290 | 291 | 292 | ## simulate unsubscribe 293 | def rem_symbol( jo ): 294 | global addrem_list 295 | 296 | jr = copy.deepcopy( jo ) 297 | sym = jr['arg'] 298 | 299 | if sym not in addrem_list: 300 | jr['code'] = 400 301 | jr['arg'] = sym + " not subscribed" 302 | else: 303 | addrem_list.remove( sym ) 304 | jr['code'] = 200 305 | jr['arg'] = sym + " unsubcribed ok" 306 | 307 | return json.dumps( jr, separators=(',', ':') ) 308 | 309 | 310 | async def broadcast( message ): 311 | PUBSUB.publish( message ) 312 | 313 | 314 | def r(l=1,u=9): return random.randint(l, u) 315 | 316 | 317 | async def broadcast_messages_count(): 318 | global stop_threads 319 | global sleepT, tf, tCount, incSym, maXTicks 320 | global addrem_list 321 | 322 | s1=s2=s3=0 323 | v1=v2=v3=0 324 | pTm = 0 325 | try: 326 | while not stop_threads: 327 | await asyncio.sleep( sleepT ) #simulate ticks in seconds 328 | 329 | dt = datetime.datetime.now() 330 | 331 | t = dt.hour*10000 + int( dt.minute/tf )*tf*100 332 | d = int( dt.strftime('%Y%m%d') ) 333 | 334 | if( pTm != t): 335 | v1 =r(3,5); v2 =r(2,3); v3 =r(1,2); pTm = t; # bar vol reset 336 | if( incSym and tCount <= maXTicks): 337 | tCount += 1; print(tCount) 338 | 339 | else: v1+=r(3,5); v2+=r(2,3); v3+=r(1,2) # bar vol cum 340 | 341 | s1+=v1; s2+=v2; s3+=v3 #total vol 342 | 343 | ## Open intentionally kept random, SYM2 test bad symbol 344 | 345 | #data = [] 346 | ##'n', 'd', 't', 'o', 'h', 'l', 'c', 'v', 'oi', 's','pc','bs','bp','as','ap' (s=total vol, pc=prev day close bs,bp,as,ap=bid ask ) 347 | data = [{"n": "SYM1", "t":t, "d":d, "c": r(1,9), "o": r(1,9), "h": 9, "l": 1, "v": v1, "oi": 0, "bp": r(1,5), "ap": r(5,9), "s": s1,"bs":1,"as":1,"pc":1,"do":4,"dh":9,"dl":1} 348 | ,{"n": "", "t":t, "d":d, "c": r(10,19), "o": r(10,19), "h": 19, "l": 10, "v": v2, "oi": 0, "bp": r(10,15), "ap": r(15,19), "s": s2,"pc":10,"do":15,"dh":19,"dl":10} 349 | ,{"n": "SYM3", "t":t, "d":d, "c": r(20,29), "o": r(20,29), "h": 29, "l": 20, "v": v3, "oi": 0, "bp": r(20,25), "ap": r(25,29), "s": s3,"pc":22,"do":28,"dh":29,"dl":20}] 350 | 351 | ## Random symbol generator code 352 | k = 4 353 | while k <= min( tCount, maXTicks): 354 | rec = {"n": "SYM"+str(k), "t":t, "d":d, "c": 18+r(), "o": 20+r(), "h": 40-r(), "l": 10+r(), "v": v1, "oi": 0, "bp": r(1,5), "ap": r(5,9), "s": s1,"bs":1,"as":1,"pc":1,"do":20,"dh":40,"dl":10} 355 | data.append( rec ) 356 | k +=1 357 | 358 | ## make ticks for subscribed symbols 359 | for asym in addrem_list: 360 | rec = {"n": asym, "t":t, "d":d, "c": 18+r(), "o": 20+r(), "h": 40-r(), "l": 10+r(), "v": v1, "oi": 0, "bp": r(1,5), "ap": r(5,9), "s": s1,"bs":1,"as":1,"pc":1,"do":20,"dh":40,"dl":10} 361 | data.append( rec ) 362 | 363 | #print( json.dumps( data, separators=(',', ':')) ) 364 | await broadcast( json.dumps( data, separators=(',', ':'))) ## remove space else plugin will not match str 365 | 366 | except asyncio.CancelledError: #raised when asyncio receives SIGINT from KB_Interrupt 367 | stop_threads = True 368 | print(f"asyncio tasks: send stop signal, wait for exit...") 369 | #try: 370 | # await asyncio.get_running_loop().shutdown_asyncgens() 371 | #except RuntimeError: pass 372 | try: 373 | await asyncio.sleep( 3 ) # these are not graceful exits. 374 | await asyncio.get_running_loop().stop() # unrecco = asyncio.get_event_loop().stop() 375 | except: pass 376 | 377 | 378 | 379 | async def start_ws_server( aport ): 380 | global tCount, incSym 381 | 382 | print( f"Started RTD server: port={aport}, tf={tf}min, sym_count={tCount}, increment_sym={incSym}" ) 383 | 384 | async with websockets.serve( handler, "localhost", aport ): 385 | await broadcast_messages_count() 386 | 387 | return 388 | 389 | 390 | async def main(): 391 | global wsport #wsport1,2,3 392 | await asyncio.gather( start_ws_server( wsport ) ) ## more tasks 393 | 394 | if __name__ == "__main__": 395 | try: sleepT = float(sys.argv[1]); print(f'Frequency={sleepT} secs') 396 | except: sleepT = 0.9; print(f'Frequency={sleepT} secs') 397 | 398 | try: 399 | print(f"### press ctrl+c to exit ###") 400 | 401 | asyncio.run( main() ) 402 | 403 | print(f"Exit 0") 404 | 405 | except KeyboardInterrupt: 406 | print(f"Kill signal, Exit 1") 407 | stop_threads = True 408 | 409 | except asyncio.CancelledError: pass 410 | except Exception: pass -------------------------------------------------------------------------------- /Utility Scripts/Python/json_validator.py: -------------------------------------------------------------------------------- 1 | ''' 2 | // ##### json_validator.py ##### 3 | // 4 | // Python utility Script to validate JSON RTD String 5 | // For Historical data, use DebugView. 6 | // 7 | // Performace penalty for Plugin to validate every permutation is unnecessary, 8 | // as a well written CLIENT=APP already performs checks on RTD. 9 | // 10 | /////////////////////////////////////////////////////////////////////// 11 | // Author: NSM51 12 | // https://github.com/ideepcoder/Rtd_Ws_AB_plugin/ 13 | // https://forum.amibroker.com/u/nsm51/summary 14 | // 15 | // Any use of this source code must include the above notice, 16 | // in the user documentation and internal comments to the code. 17 | ''' 18 | 19 | 20 | import json 21 | 22 | ## Insert RTD json string in '' quotes to validate 23 | js = '[{"n":"SYM3","d":20250127,"t":230908,"o":189.45,"h":196.32,"l":181.89,"c":194.59,"v":1212,"bp":186.71,"as":10},\ 24 | {"n":"SYM2","d":20250127,"t":230908,"o":189.45,"h":196.32,"l":181.89,"c":194.59,"v":1212,"ap":186.71,"bs":10}]' 25 | 26 | ############### 27 | # Fields used by RTD json 28 | 29 | Field_str = [ 'n' ] 30 | Field_num_reqd = [ 'o', 'h', 'l', 'c' ] 31 | Field_num_optional = [ 'v', 'oi', 'x1', 'x2', 's', 'pc', 'bs', 'bp', 'as', 'ap', 'do', 'dh', 'dl' ] 32 | 33 | 34 | def check_str_reqd( f, r ): 35 | if f in r: 36 | if not isinstance( r[f], str ): 37 | print( f"{f} not str in \n{r}" ); return False 38 | else: 39 | print( f"{f} not found in \n{r}" ); return False 40 | 41 | return True 42 | 43 | 44 | def check_num_reqd( f, r ): 45 | if f in r: 46 | if not isinstance( r[f], (int, float) ): 47 | print( f"{f} not NUM in \n{r}" ); return False 48 | else: 49 | print( f"{f} not found in \n{r}" ); return False 50 | 51 | return True 52 | 53 | 54 | def check_num_opt( f, r ): 55 | if f in r: 56 | if not isinstance( r[f], (int, float) ): 57 | print( f"{f} not NUM in \n{r}" ); return False 58 | 59 | return True 60 | 61 | 62 | ''' 63 | Function to validate type of mandatory & optional fields 64 | ''' 65 | def json_rtd_validate( js:str ): 66 | 67 | if js.startswith('[{"n"'): print(f"\nString MATCH = ok") 68 | else: print(f"\nString MATCH failed"); return 69 | 70 | try: 71 | jo = json.loads( js ) 72 | print( f"Json parsing = ok" ) 73 | except Exception as e: 74 | print( e ); return 75 | 76 | 77 | if isinstance( jo, list): 78 | print("Outer Array = ok\n") 79 | else: 80 | print( f"Not Array of Quotes"); return 81 | 82 | 83 | i = 0; 84 | for r in jo: 85 | 86 | if not isinstance( r, dict ): print(f"record should be dict"); return 87 | 88 | ## test mandatory 'n' and type 89 | if not check_str_reqd( 'n', r ): return 90 | 91 | ## OHLC, test mandatory and type 92 | for ro in Field_num_reqd: 93 | if not check_num_reqd( ro, r ): return 94 | 95 | ## Optionals, test type 96 | for ro in Field_num_optional: 97 | if not check_num_opt( ro, r ): return 98 | 99 | ## Date 100 | if 'd' in r: 101 | if type( r['d'] ) is not int: 102 | print( f"d not int \n{r}" ); return 103 | 104 | if 't' in r: 105 | if type( r['t'] ) is not int : 106 | print( f"t not int \n{r}" ); return 107 | else: 108 | print( f"t is required with d in \n{r}"); return 109 | 110 | elif 'u' in r: 111 | if type( r['u'] ) is not int : 112 | print( f"u not int \n{r}" ); return 113 | 114 | elif 'g' in r: 115 | if type( r['g'] ) is not int : 116 | print( f"g not int \n{r}" ); return 117 | 118 | else: print( f"d/u/g not found in \n{r}" ); return 119 | 120 | print( f"record {i} = ok" ); i += 1 121 | 122 | return 1 123 | 124 | 125 | #### main() 126 | if __name__ == '__main__': 127 | 128 | if json_rtd_validate( js ): 129 | print( f"\nFinal validation = OK\n") 130 | else: 131 | print( f"\nFinal validation = FAILED\n" ) 132 | 133 | #print( js ) ## print original string 134 | 135 | -------------------------------------------------------------------------------- /Utility Scripts/Python/notice.txt: -------------------------------------------------------------------------------- 1 | This plugin should be distributed or redisributed with the Documentation 2 | file provided in HTML that contains all the Licence information, 3 | Trademark, Copyright & Other notices. -------------------------------------------------------------------------------- /google04cf242d3a5a7326.html: -------------------------------------------------------------------------------- 1 | google-site-verification: google04cf242d3a5a7326.html -------------------------------------------------------------------------------- /images/build.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ideepcoder/Rtd_Ws_AB_plugin/90dc7e12bf954272d6299bbca50509521b63c1ce/images/build.png -------------------------------------------------------------------------------- /images/help/DB_Create.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ideepcoder/Rtd_Ws_AB_plugin/90dc7e12bf954272d6299bbca50509521b63c1ce/images/help/DB_Create.png -------------------------------------------------------------------------------- /images/help/Plugin_Configure.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ideepcoder/Rtd_Ws_AB_plugin/90dc7e12bf954272d6299bbca50509521b63c1ce/images/help/Plugin_Configure.png -------------------------------------------------------------------------------- /images/help/Plugin_baloon_OK.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ideepcoder/Rtd_Ws_AB_plugin/90dc7e12bf954272d6299bbca50509521b63c1ce/images/help/Plugin_baloon_OK.png -------------------------------------------------------------------------------- /images/help/Plugin_menu.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ideepcoder/Rtd_Ws_AB_plugin/90dc7e12bf954272d6299bbca50509521b63c1ce/images/help/Plugin_menu.png -------------------------------------------------------------------------------- /images/help/RT_Data_index.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ideepcoder/Rtd_Ws_AB_plugin/90dc7e12bf954272d6299bbca50509521b63c1ce/images/help/RT_Data_index.png -------------------------------------------------------------------------------- /images/help/Retrieve.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ideepcoder/Rtd_Ws_AB_plugin/90dc7e12bf954272d6299bbca50509521b63c1ce/images/help/Retrieve.png -------------------------------------------------------------------------------- /images/help/TJP_Registry.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ideepcoder/Rtd_Ws_AB_plugin/90dc7e12bf954272d6299bbca50509521b63c1ce/images/help/TJP_Registry.png -------------------------------------------------------------------------------- /images/help/debug1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ideepcoder/Rtd_Ws_AB_plugin/90dc7e12bf954272d6299bbca50509521b63c1ce/images/help/debug1.png -------------------------------------------------------------------------------- /images/help/sample_s1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ideepcoder/Rtd_Ws_AB_plugin/90dc7e12bf954272d6299bbca50509521b63c1ce/images/help/sample_s1.png -------------------------------------------------------------------------------- /images/help/sample_s2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ideepcoder/Rtd_Ws_AB_plugin/90dc7e12bf954272d6299bbca50509521b63c1ce/images/help/sample_s2.png -------------------------------------------------------------------------------- /images/help/workflow_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ideepcoder/Rtd_Ws_AB_plugin/90dc7e12bf954272d6299bbca50509521b63c1ce/images/help/workflow_1.png -------------------------------------------------------------------------------- /images/help/workflow_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ideepcoder/Rtd_Ws_AB_plugin/90dc7e12bf954272d6299bbca50509521b63c1ce/images/help/workflow_2.png -------------------------------------------------------------------------------- /images/unknown.svg: -------------------------------------------------------------------------------- 1 | buildbuildunknownunknown --------------------------------------------------------------------------------