├── README.md ├── commands.txt ├── delta_spark_jupyter ├── __init__.py ├── delta_spark.Dockerfile ├── requirements.txt └── src │ ├── exec.ipynb │ ├── get_schema-registry.ipynb │ ├── producer_crypto.ipynb │ ├── spark-sql-kafka-0-10_2.12-3.1.2.jar │ ├── spark-streaming-kafka-0-10_2.13-3.2.1.jar │ └── spark_streaming_delta.ipynb ├── docker-compose.yml ├── docs ├── add.png ├── bar.png ├── config.png ├── creden.png ├── data.png ├── hive.png ├── pbi.png ├── stak_bigdata_arq.svg └── success.png ├── hive_presto ├── hadoop-hive.env └── presto │ ├── catalog │ └── hive.properties │ ├── config.properties │ ├── jvm.config │ └── node.properties ├── hue └── config │ └── hue.ini ├── minio └── minio.env ├── producer ├── .dockerignore ├── __init__.py ├── __pycache__ │ └── map_avro_types.cpython-38.pyc ├── avro │ └── crypto-schema.avsc ├── crypto_producer.py ├── map_avro_types.py ├── producer_confs.env ├── producer_service.Dockerfile └── producer_service_requirements.txt └── superset └── config └── superset_config.py /README.md: -------------------------------------------------------------------------------- 1 | # Big Data Streaming Stack 2 | 3 | ![Example](./docs/stak_bigdata_arq.svg) 4 | 5 | Big Data Streaming Stack es un conjunto de tecnologías unificadas con Docker Compose cuyo 6 | proposito es facilitar el aprendizaje de algunas tecnologías como Delta Lake en modo on-premise. 7 | 8 | ## Dependencias 9 | 10 | - Docker 20.10.7 11 | - Docker Compose 1.29.2 12 | 13 | **Nota**: Se recomienda usar docker en alguna distribución de linux para mayor agilidad al momento 14 | de instalar el stack. 15 | 16 | ## Inicio Rápido 17 | 18 | Clonar o descargar el repositorio: 19 | 20 | `git clone https://github.com/santiagomj/Big-Data-Streaming-Stack.git` 21 | 22 | Acceder por consola o algún ambiente de desarrollo a la carpeta donde fue descargado 23 | el repositorio y ejecutar docker compose: 24 | 25 | `docker-compose up` 26 | 27 | Creación de usuario Apache Superset 28 | 29 | `docker exec -it superset superset-init` 30 | 31 | ## Servicios locales 32 | 33 | - Control Center: 34 | - Jupyter Notebook: 35 | - Hue: 36 | - Superset: 37 | - Minio: 38 | 39 | ## Conexión con Power BI 📊 y Hive SQL 🐝 40 | 41 | Este apartado es para aquellos usuarios que no desean usar super set 42 | si no power BI, para ello debemos hacer unas configuraciones adicionales: 43 | 44 | ### Instalación del ODBC (Open DataBase Connectivity) 45 | 46 | Esta utilidad nos servirá para conectarnos a cualquier 47 | base de datos sin importar el gestor que esté maejando la 48 | aplicación. 49 | 50 | Procedemos a descargar la utlidad en el siguiente link: 51 | 52 | - 53 | 54 | Le damos click en descargar y e instalamos, luego en windows nos vamos para la barra de tareas y en la barra de busqueda colocamos: 55 | 56 | "Origenes de datos ODBC (64 bits)" 57 | 58 | ![barra](./docs/bar.png) 59 | 60 | Una vez abierto la aplicacion le damos en add... o agregar: 61 | 62 | ![add](./docs/add.png) 63 | 64 | Nos saldrá un cuadro con los drivers que tenemos disponibles para 65 | realiazar la conexión, debemos de fijarnos en el que diga 66 | **Microsoft Hive ODBC driver** y le damos click en finalizar. 67 | 68 | Acontinuación nos aparecerá una ventaja con estas caracteristicas: 69 | 70 | ![config](./docs/config.png) 71 | 72 | Los campos que nos interesan son: 73 | 74 | - **Data Source Name:** el cual tendrá el nombre de nuestra conexión, es 75 | recomendable que sea memotecnico para diferenciar los diferentes tipos de conexiones. 76 | 77 | Para este caso le colocamos **Hive_Connection**. 78 | 79 | - **Host(s):** Allí irá el host donde está localizado tu servicio de bases de datos. 80 | 81 | Como el proyecto se está trabajando en docker, es importante tener en cuenta donde está ubicado el host del servcio y el puerto. 82 | 83 | en este caso el host es: **localhost** y el puerto es: **10000**. 84 | 85 | - **Database:** Aqui va el nombre de la Base de datos, por defecto es default, sin embargo si tienen una base de datos con un nombre diferente este debe de ir en este apartado. 86 | 87 | - **Authentication/Mechanism:** Si su servicio posee algun tipo de autenticación especifica se debe de seleccionar alguna de las opciones con la se está auntenticando tu servicio. 88 | 89 | Para este caso es : **Username** 90 | 91 | Los demas campos se dejan en blanco y procedemos a realizar click en **test** 92 | 93 | si la conexión fue exitosa nos aparecerá un cuadro de texto con un **succes!** 94 | 95 | ![success](./docs/success.png) 96 | 97 | Hecho esto le damos en **Ok** y **Aceptar**, el gestor se cierra automaticamente. 98 | 99 | ### Conexión con power BI Y OBDC 📊 100 | 101 | Procedemos con la creación de un nuevo proyecto en Power BI desktop. 102 | 103 | Luego le damos click a obtener Datos, Otras, buscamos OBDC y conectar: 104 | 105 | ![powerbi](./docs/pbi.png) 106 | 107 | Nos aparecerá un cuadro pidiendonos usuario y contraseña. 108 | Para ello es importante revisar en el [docker-compose.yml](docker-compose.yml) en el servicio llamado hive-server, 109 | encuentras unas variables de entorno llamadas **HDFS_CONF_fs_s3a** en ellas encuentras una con **access_key** y **secret_key**, las cuales juegan el papel del usuario y password respectivamente, con esta información puedes realizar la conexión. 110 | 111 | - **username**: crypto-prices. 112 | - **password**: crypto-prices. 113 | 114 | ![creden](./docs/creden.png) 115 | 116 | **IMPORTANTE**: Las tablas que se ven a continuación, no estan definidas inicialmente en el Docker Compose, por lo que inicialemnte, no existirá ninguna tabla en la base de datos "default". dichas tablas se pueden crear con Apache Hive (Desde la interfaz de Hue) como tablas externas, en el archivo commands.txt se encuentran algunos comandos que pueden ser de ayuda para crear tablas externas. 117 | 118 | No saldrá el navegador, desde allí nos dirijimos a **HIVE**, **default** o el nombre de tu base de datos, **selecionas** las tablas que deseas trabajar y finalmente en **cargar**, tendrás tus datos cargados y listos para trabajar con **power BI**. 119 | 120 | ![data](./docs/data.png) 121 | -------------------------------------------------------------------------------- /commands.txt: -------------------------------------------------------------------------------- 1 | kafka-topics.sh --create --zookeeper zookeeper:2181 --replication-factor 1 --partitions 1 --topic crypto-prices 2 | kafka-topics.sh --list --zookeeper zookeeper:2181 3 | kafka-topics.sh --delete --zookeeper zookeeper:2181 --topic crypto-prices 4 | 5 | kafka-console-producer.sh --broker-list kafka:9092 --topic crypto-prices 6 | {'user_id': 1, 'recipient_id': 2, 'message': 'Hi'} 7 | {'user_id': 2, 'recipient_id': 1, 'message': 'Hello there'} 8 | 9 | kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic crypto-prices --from-beginning 10 | kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic crypto-prices 11 | 12 | kafka-topics.sh --version 13 | 14 | find . -type f -name "kafka-topics.sh" 15 | ./opt/kafka_2.13-2.8.1/bin/kafka-topics.sh 16 | 17 | find . -type f -name "spark-submit" 18 | ./local/bin/spark-submit 19 | 20 | find . -type f -name "server.properties" 21 | ./opt/kafka_2.13-2.8.1/config/server.properties 22 | 23 | [root@hadoop-master /]# hive 24 | hive> use default; 25 | hive> create external table customer_text(id int, id_usuario int, id_rank int) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS TEXTFILE location 's3a://crypto-prices2/'; 26 | hive> select * from customer_text; 27 | 28 | # find port 29 | set hive.server2.thrift.port; 30 | 31 | docker exec -it superset superset-init 32 | 33 | hive://hive@hive-server:10000/default 34 | 35 | CREATE TABLE cryto_prices ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' STORED AS AVRO TBLPROPERTIES ('avro.schema.url'='s3a://avro-schema/crypto-schema.avsc'); 36 | CREATE EXTERNAL TABLE cryto_prices2 LIKE cryto_prices STORED AS PARQUET LOCATION 's3a://crypto-prices/events'; 37 | select * from cryto_prices2; 38 | 39 | 40 | 41 | -------------------------------------------------------------------------------- /delta_spark_jupyter/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/santiagomj/Big-Data-Streaming-Stack/d163ff9eea9c6ce7ff013cc39c6431b59f9bea3e/delta_spark_jupyter/__init__.py -------------------------------------------------------------------------------- /delta_spark_jupyter/delta_spark.Dockerfile: -------------------------------------------------------------------------------- 1 | FROM jupyter/pyspark-notebook:8ea7abc5b7bc 2 | USER root 3 | ENV PYSPARK_SUBMIT_ARGS '--packages org.apache.spark:spark-avro_2.12:3.0.1,org.apache.spark:spark-sql-kafka-0-10_2.12:3.0.1,org.apache.spark:spark-streaming-kafka-0-10_2.12:3.0.1,io.delta:delta-core_2.12:1.1.0,com.amazonaws:aws-java-sdk:1.11.950,org.apache.hadoop:hadoop-aws:3.2.0,net.java.dev.jets3t:jets3t:0.9.4 pyspark-shell' 4 | 5 | COPY ./delta_spark_jupyter /src 6 | WORKDIR /src 7 | #spark 8 | ARG DEBIAN_FRONTEND=noninteractive 9 | 10 | RUN apt-get update && apt-get install build-essential -y 11 | RUN apt-get update && apt-get install -y python3-pip python3-dev 12 | RUN apt-get install -y curl 13 | RUN curl -O https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/1.11.950/aws-java-sdk-bundle-1.11.950.jar 14 | RUN curl -O https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/3.2.0/hadoop-aws-3.2.0.jar 15 | RUN curl -O https://repo1.maven.org/maven2/net/java/dev/jets3t/jets3t/0.9.4/jets3t-0.9.4.jar 16 | 17 | RUN \ 18 | apt-get update && \ 19 | apt-get install -y openjdk-8-jdk && \ 20 | rm -rf /var/lib/apt/lists/* 21 | 22 | RUN apt-get update && apt-get install software-properties-common -y 23 | 24 | RUN wget -qO - http://packages.confluent.io/deb/3.2/archive.key | apt-key add - 25 | 26 | RUN add-apt-repository "deb [arch=amd64] http://packages.confluent.io/deb/3.2 stable main" 27 | RUN apt-get update && apt-get install confluent-platform-oss-2.11 -y 28 | 29 | RUN apt-get update && apt install librdkafka-dev -y 30 | 31 | RUN pip install -r requirements.txt 32 | 33 | CMD ["bash"] -------------------------------------------------------------------------------- /delta_spark_jupyter/requirements.txt: -------------------------------------------------------------------------------- 1 | delta-spark==1.1.0 2 | confluent-kafka[avro]==1.4.0 -------------------------------------------------------------------------------- /delta_spark_jupyter/src/exec.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": { 7 | "tags": [] 8 | }, 9 | "outputs": [ 10 | { 11 | "name": "stdout", 12 | "output_type": "stream", 13 | "text": [ 14 | "Producing message @ 2022-02-18 03:55:14.226061 | Message = {'user_id': 94, 'recipient_id': 100, 'message': 'djpMwZPilAnYwEplHcyTLvYyesBHTwjm'}\n", 15 | "Producing message @ 2022-02-18 03:55:20.234221 | Message = {'user_id': 63, 'recipient_id': 37, 'message': 'ibmrvBRbHjgVPeCeTDiLnrQZurAgBJVF'}\n", 16 | "Producing message @ 2022-02-18 03:55:28.242006 | Message = {'user_id': 9, 'recipient_id': 44, 'message': 'TolTyncOUoFQdbUmCSUcHCGHjtdfrLHl'}\n", 17 | "Producing message @ 2022-02-18 03:55:34.246182 | Message = {'user_id': 100, 'recipient_id': 21, 'message': 'rgYNyzNPcHplKqlvqipRGAwEkgBUHDpW'}\n", 18 | "Producing message @ 2022-02-18 03:55:38.251099 | Message = {'user_id': 80, 'recipient_id': 53, 'message': 'WiPPnJXrJNpvQFPaChmfmsGIgtYkAklf'}\n", 19 | "Producing message @ 2022-02-18 03:55:45.258171 | Message = {'user_id': 2, 'recipient_id': 7, 'message': 'XTPIogwqzQcSLUMkzZiMxOUZqXpVkDHY'}\n", 20 | "Producing message @ 2022-02-18 03:55:52.262151 | Message = {'user_id': 32, 'recipient_id': 97, 'message': 'GJtYbBrsFhpLnUvZbmqZhxCDkMpxjVTC'}\n", 21 | "Producing message @ 2022-02-18 03:55:55.266136 | Message = {'user_id': 87, 'recipient_id': 62, 'message': 'ScGeAzdRPvpxwGgQBRACCqojqqVaJudQ'}\n" 22 | ] 23 | } 24 | ], 25 | "source": [ 26 | "!python3 producer.py" 27 | ] 28 | }, 29 | { 30 | "cell_type": "code", 31 | "execution_count": 12, 32 | "metadata": {}, 33 | "outputs": [ 34 | { 35 | "name": "stdout", 36 | "output_type": "stream", 37 | "text": [ 38 | "Successfully producing record value - {'user_id': 1, 'recipient_id': 1, 'message': 'GG!'} to topic - messages\n" 39 | ] 40 | } 41 | ], 42 | "source": [ 43 | "!python3 send_record.py --topic messages --schema-file create-user-request.avsc --record-value '{\"user_id\": 1, \"recipient_id\": 1, \"message\": \"GG!\"}'" 44 | ] 45 | }, 46 | { 47 | "cell_type": "code", 48 | "execution_count": null, 49 | "metadata": {}, 50 | "outputs": [ 51 | { 52 | "name": "stdout", 53 | "output_type": "stream", 54 | "text": [ 55 | "Successfully producing record values - count: 100 to topic - test\n", 56 | "Successfully producing record values - count: 100 to topic - test\n", 57 | "Successfully producing record values - count: 100 to topic - test\n", 58 | "Successfully producing record values - count: 100 to topic - test\n", 59 | "Successfully producing record values - count: 100 to topic - test\n" 60 | ] 61 | } 62 | ], 63 | "source": [ 64 | "!cd producer/ && python3 crypto_producer.py" 65 | ] 66 | }, 67 | { 68 | "cell_type": "code", 69 | "execution_count": null, 70 | "metadata": {}, 71 | "outputs": [], 72 | "source": [ 73 | "if response.status_code == 200:\n", 74 | " data = json.loads(response.text)[\"data\"]\n", 75 | "else:\n", 76 | " print(f\"Request Exception {status}\")" 77 | ] 78 | } 79 | ], 80 | "metadata": { 81 | "kernelspec": { 82 | "display_name": "Python 3", 83 | "language": "python", 84 | "name": "python3" 85 | }, 86 | "language_info": { 87 | "codemirror_mode": { 88 | "name": "ipython", 89 | "version": 3 90 | }, 91 | "file_extension": ".py", 92 | "mimetype": "text/x-python", 93 | "name": "python", 94 | "nbconvert_exporter": "python", 95 | "pygments_lexer": "ipython3", 96 | "version": "3.8.6" 97 | } 98 | }, 99 | "nbformat": 4, 100 | "nbformat_minor": 5 101 | } 102 | -------------------------------------------------------------------------------- /delta_spark_jupyter/src/get_schema-registry.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [ 8 | { 9 | "data": { 10 | "text/plain": [ 11 | "'{\"type\":\"record\",\"name\":\"CryptoSchema\",\"namespace\":\"io.codebrews.cryptoschema\",\"fields\":[{\"name\":\"id\",\"type\":\"string\"},{\"name\":\"rank\",\"type\":\"int\"},{\"name\":\"symbol\",\"type\":\"string\"},{\"name\":\"name\",\"type\":\"string\"},{\"name\":\"supply\",\"type\":[\"float\",\"null\"]},{\"name\":\"maxSupply\",\"type\":[\"float\",\"null\"]},{\"name\":\"marketCapUsd\",\"type\":[\"float\",\"null\"]},{\"name\":\"volumeUsd24Hr\",\"type\":[\"float\",\"null\"]},{\"name\":\"priceUsd\",\"type\":[\"float\",\"null\"]},{\"name\":\"changePercent24Hr\",\"type\":[\"float\",\"null\"]},{\"name\":\"vwap24Hr\",\"type\":[\"float\",\"null\"]},{\"name\":\"explorer\",\"type\":[\"string\",\"null\"]}]}'" 12 | ] 13 | }, 14 | "execution_count": 1, 15 | "metadata": {}, 16 | "output_type": "execute_result" 17 | } 18 | ], 19 | "source": [ 20 | "# Retrieve GPS Schema grom Schema Registry\n", 21 | "\n", 22 | "from confluent_kafka.schema_registry import SchemaRegistryClient\n", 23 | "\n", 24 | "schema_registry_conf = {'url': \"http://schema-registry:8081\"}\n", 25 | "\n", 26 | "\n", 27 | "schema_registry_client = SchemaRegistryClient(schema_registry_conf)\n", 28 | "schema_response = schema_registry_client.get_latest_version(\"crypto-prices\" + \"-value\").schema\n", 29 | "schema = schema_response.schema_str\n", 30 | "schema" 31 | ] 32 | }, 33 | { 34 | "cell_type": "code", 35 | "execution_count": 3, 36 | "metadata": {}, 37 | "outputs": [ 38 | { 39 | "name": "stdout", 40 | "output_type": "stream", 41 | "text": [ 42 | "Reading package lists... Done\n", 43 | "Building dependency tree \n", 44 | "Reading state information... Done\n", 45 | "The following NEW packages will be installed:\n", 46 | " iputils-ping\n", 47 | "0 upgraded, 1 newly installed, 0 to remove and 125 not upgraded.\n", 48 | "Need to get 40.1 kB of archives.\n", 49 | "After this operation, 108 kB of additional disk space will be used.\n", 50 | "Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 iputils-ping amd64 3:20190709-3 [40.1 kB]\n", 51 | "Fetched 40.1 kB in 1s (72.1 kB/s) \u001b[0m\u001b[33m\n", 52 | "debconf: delaying package configuration, since apt-utils is not installed\n", 53 | "\n", 54 | "\u001b7\u001b[0;23r\u001b8\u001b[1ASelecting previously unselected package iputils-ping.\n", 55 | "(Reading database ... 68529 files and directories currently installed.)\n", 56 | "Preparing to unpack .../iputils-ping_3%3a20190709-3_amd64.deb ...\n", 57 | "\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 0%]\u001b[49m\u001b[39m [..........................................................] \u001b8\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 20%]\u001b[49m\u001b[39m [###########...............................................] \u001b8Unpacking iputils-ping (3:20190709-3) ...\n", 58 | "\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 40%]\u001b[49m\u001b[39m [#######################...................................] \u001b8Setting up iputils-ping (3:20190709-3) ...\n", 59 | "\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 60%]\u001b[49m\u001b[39m [##################################........................] \u001b8\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 80%]\u001b[49m\u001b[39m [##############################################............] \u001b8\n", 60 | "\u001b7\u001b[0;24r\u001b8\u001b[1A\u001b[J" 61 | ] 62 | } 63 | ], 64 | "source": [ 65 | "!apt install iputils-ping" 66 | ] 67 | }, 68 | { 69 | "cell_type": "code", 70 | "execution_count": 2, 71 | "metadata": {}, 72 | "outputs": [ 73 | { 74 | "name": "stdout", 75 | "output_type": "stream", 76 | "text": [ 77 | "/bin/bash: ping: command not found\n" 78 | ] 79 | } 80 | ], 81 | "source": [ 82 | "!ping minio" 83 | ] 84 | }, 85 | { 86 | "cell_type": "code", 87 | "execution_count": 2, 88 | "metadata": {}, 89 | "outputs": [ 90 | { 91 | "name": "stdout", 92 | "output_type": "stream", 93 | "text": [ 94 | "/usr/local/spark/python/pyspark/__init__.py\n" 95 | ] 96 | } 97 | ], 98 | "source": [ 99 | "import pyspark\n", 100 | "print(pyspark.__file__)" 101 | ] 102 | }, 103 | { 104 | "cell_type": "code", 105 | "execution_count": 6, 106 | "metadata": { 107 | "collapsed": true, 108 | "jupyter": { 109 | "outputs_hidden": true 110 | }, 111 | "tags": [] 112 | }, 113 | "outputs": [ 114 | { 115 | "name": "stdout", 116 | "output_type": "stream", 117 | "text": [ 118 | "Collecting package metadata (current_repodata.json): done\n", 119 | "Solving environment: done\n", 120 | "\n", 121 | "## Package Plan ##\n", 122 | "\n", 123 | " environment location: /opt/conda\n", 124 | "\n", 125 | " added / updated specs:\n", 126 | " - python-confluent-kafka\n", 127 | "\n", 128 | "\n", 129 | "The following packages will be downloaded:\n", 130 | "\n", 131 | " package | build\n", 132 | " ---------------------------|-----------------\n", 133 | " cyrus-sasl-2.1.27 | h230043b_5 228 KB conda-forge\n", 134 | " gettext-0.19.8.1 | h73d1719_1008 3.6 MB conda-forge\n", 135 | " libgcrypt-1.10.0 | h7f98852_0 705 KB conda-forge\n", 136 | " libgpg-error-1.44 | h9eb791d_0 294 KB conda-forge\n", 137 | " libgsasl-1.10.0 | h5b4c23d_0 179 KB conda-forge\n", 138 | " libntlm-1.4 | h7f98852_1002 32 KB conda-forge\n", 139 | " librdkafka-1.7.0 | hc49e61c_1 12.9 MB conda-forge\n", 140 | " python-confluent-kafka-1.7.0| py39h3811e60_2 132 KB conda-forge\n", 141 | " ------------------------------------------------------------\n", 142 | " Total: 18.0 MB\n", 143 | "\n", 144 | "The following NEW packages will be INSTALLED:\n", 145 | "\n", 146 | " cyrus-sasl conda-forge/linux-64::cyrus-sasl-2.1.27-h230043b_5\n", 147 | " gettext conda-forge/linux-64::gettext-0.19.8.1-h73d1719_1008\n", 148 | " libgcrypt conda-forge/linux-64::libgcrypt-1.10.0-h7f98852_0\n", 149 | " libgpg-error conda-forge/linux-64::libgpg-error-1.44-h9eb791d_0\n", 150 | " libgsasl conda-forge/linux-64::libgsasl-1.10.0-h5b4c23d_0\n", 151 | " libntlm conda-forge/linux-64::libntlm-1.4-h7f98852_1002\n", 152 | " librdkafka conda-forge/linux-64::librdkafka-1.7.0-hc49e61c_1\n", 153 | " python-confluent-~ conda-forge/linux-64::python-confluent-kafka-1.7.0-py39h3811e60_2\n", 154 | "\n", 155 | "\n", 156 | "\n", 157 | "Downloading and Extracting Packages\n", 158 | "libgcrypt-1.10.0 | 705 KB | ##################################### | 100% \n", 159 | "gettext-0.19.8.1 | 3.6 MB | ##################################### | 100% \n", 160 | "cyrus-sasl-2.1.27 | 228 KB | ##################################### | 100% \n", 161 | "python-confluent-kaf | 132 KB | ##################################### | 100% \n", 162 | "libgsasl-1.10.0 | 179 KB | ##################################### | 100% \n", 163 | "libgpg-error-1.44 | 294 KB | ##################################### | 100% \n", 164 | "librdkafka-1.7.0 | 12.9 MB | ##################################### | 100% \n", 165 | "libntlm-1.4 | 32 KB | ##################################### | 100% \n", 166 | "Preparing transaction: done\n", 167 | "Verifying transaction: done\n", 168 | "Executing transaction: done\n" 169 | ] 170 | } 171 | ], 172 | "source": [ 173 | "!conda install -c conda-forge python-confluent-kafka -y" 174 | ] 175 | }, 176 | { 177 | "cell_type": "code", 178 | "execution_count": null, 179 | "metadata": {}, 180 | "outputs": [], 181 | "source": [ 182 | "...\n", 183 | " environment:\n", 184 | " - PYSPARK_SUBMIT_ARGS=--packages com.amazonaws:aws-java-sdk-bundle:1.11.819,org.apache.hadoop:hadoop-aws:3.2.0 pyspark-shell\n", 185 | "..." 186 | ] 187 | }, 188 | { 189 | "cell_type": "code", 190 | "execution_count": null, 191 | "metadata": {}, 192 | "outputs": [], 193 | "source": [ 194 | "# os.environ['PYSPARK_SUBMIT_ARGS'] = \"\"\"--packages org.apache.spark:spark-avro_2.12:3.2.1,org.apache.spark:spark-sql-kafka-0-10_2.12:3.1.2,org.apache.spark:spark-streaming-kafka-0-10_2.13:3.2.1,io.delta:delta-core_2.12:1.1.0,com.amazonaws:aws-java-sdk:1.11.950,org.apache.hadoop:hadoop-aws:3.2.0,net.java.dev.jets3t:jets3t:0.9.4 pyspark-shell\"\"\"\n" 195 | ] 196 | } 197 | ], 198 | "metadata": { 199 | "kernelspec": { 200 | "display_name": "Python 3", 201 | "language": "python", 202 | "name": "python3" 203 | }, 204 | "language_info": { 205 | "codemirror_mode": { 206 | "name": "ipython", 207 | "version": 3 208 | }, 209 | "file_extension": ".py", 210 | "mimetype": "text/x-python", 211 | "name": "python", 212 | "nbconvert_exporter": "python", 213 | "pygments_lexer": "ipython3", 214 | "version": "3.8.6" 215 | } 216 | }, 217 | "nbformat": 4, 218 | "nbformat_minor": 5 219 | } 220 | -------------------------------------------------------------------------------- /delta_spark_jupyter/src/producer_crypto.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import requests\n", 10 | "import json\n", 11 | "import uuid\n", 12 | "import os \n", 13 | "\n", 14 | "from confluent_kafka.avro import AvroProducer\n", 15 | "from confluent_kafka import avro\n", 16 | "\n", 17 | "url = os.getenv(\"CRYPTO_API_URL_ASSETS\")\n", 18 | "\n", 19 | "payload={}\n", 20 | "headers = {}\n", 21 | "\n", 22 | "response = requests.request(\"GET\", url, headers=headers, data=payload)\n", 23 | "\n", 24 | "if response.status_code == 200:\n", 25 | " data = json.loads(response.text)[\"data\"]\n", 26 | "else:\n", 27 | " print(\"no hay conex\")" 28 | ] 29 | }, 30 | { 31 | "cell_type": "code", 32 | "execution_count": 2, 33 | "metadata": {}, 34 | "outputs": [ 35 | { 36 | "ename": "FileNotFoundError", 37 | "evalue": "[Errno 2] No such file or directory: './avro/crypto-schema.avsc'", 38 | "output_type": "error", 39 | "traceback": [ 40 | "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", 41 | "\u001b[0;31mFileNotFoundError\u001b[0m Traceback (most recent call last)", 42 | "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[1;32m 9\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 10\u001b[0m \u001b[0mkey_schema\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mavro\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mloads\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkey_schema_string\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 11\u001b[0;31m \u001b[0mvalue_schema\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mavro\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mload\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mschema_file\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", 43 | "\u001b[0;32m/opt/conda/lib/python3.8/site-packages/confluent_kafka/avro/load.py\u001b[0m in \u001b[0;36mload\u001b[0;34m(fp)\u001b[0m\n\u001b[1;32m 34\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mload\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfp\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 35\u001b[0m \u001b[0;34m\"\"\" Parse a schema from a file path \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 36\u001b[0;31m \u001b[0;32mwith\u001b[0m \u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfp\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 37\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mloads\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 38\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", 44 | "\u001b[0;31mFileNotFoundError\u001b[0m: [Errno 2] No such file or directory: './avro/crypto-schema.avsc'" 45 | ] 46 | } 47 | ], 48 | "source": [ 49 | "topic = os.getenv(\"TOPIC\")\n", 50 | "bootstrap_servers = os.getenv(\"BOOTSTRAP_SERVERS\")\n", 51 | "schema_registry = os.getenv(\"SCHEMA_REGISTRY\")\n", 52 | "schema_file = os.getenv(\"SCHEMA_FILE\")\n", 53 | "key_schema_string = os.getenv(\"KEY_SCHEMA_STRING\")\n", 54 | "# key_schema_string = \"\"\"\n", 55 | "# {\"type\": \"string\"}\n", 56 | "# \"\"\"\n", 57 | "\n", 58 | "key_schema = avro.loads(key_schema_string)\n", 59 | "value_schema = avro.load(schema_file)" 60 | ] 61 | }, 62 | { 63 | "cell_type": "code", 64 | "execution_count": 8, 65 | "metadata": { 66 | "collapsed": true, 67 | "jupyter": { 68 | "outputs_hidden": true 69 | }, 70 | "tags": [] 71 | }, 72 | "outputs": [ 73 | { 74 | "data": { 75 | "text/plain": [ 76 | "[{'id': 'bitcoin',\n", 77 | " 'rank': 1,\n", 78 | " 'symbol': 'BTC',\n", 79 | " 'name': 'Bitcoin',\n", 80 | " 'supply': 18963231.0,\n", 81 | " 'maxSupply': 21000000.0,\n", 82 | " 'marketCapUsd': 757358169325.7595,\n", 83 | " 'volumeUsd24Hr': 6620212233.287136,\n", 84 | " 'priceUsd': 39938.24519280283,\n", 85 | " 'changePercent24Hr': -0.7508464039672565,\n", 86 | " 'vwap24Hr': 39999.63645150823,\n", 87 | " 'explorer': 'https://blockchain.info/'},\n", 88 | " {'id': 'ethereum',\n", 89 | " 'rank': 2,\n", 90 | " 'symbol': 'ETH',\n", 91 | " 'name': 'Ethereum',\n", 92 | " 'supply': 119652189.249,\n", 93 | " 'maxSupply': None,\n", 94 | " 'marketCapUsd': 326059728809.07825,\n", 95 | " 'volumeUsd24Hr': 6517909807.556441,\n", 96 | " 'priceUsd': 2725.062791208422,\n", 97 | " 'changePercent24Hr': -3.2126727732464775,\n", 98 | " 'vwap24Hr': 2758.522378463274,\n", 99 | " 'explorer': 'https://etherscan.io/'},\n", 100 | " {'id': 'tether',\n", 101 | " 'rank': 3,\n", 102 | " 'symbol': 'USDT',\n", 103 | " 'name': 'Tether',\n", 104 | " 'supply': 79019501685.78094,\n", 105 | " 'maxSupply': None,\n", 106 | " 'marketCapUsd': 79403306198.17981,\n", 107 | " 'volumeUsd24Hr': 19291687473.670258,\n", 108 | " 'priceUsd': 1.0048570859624637,\n", 109 | " 'changePercent24Hr': 0.2268242011354477,\n", 110 | " 'vwap24Hr': 1.0018577389418242,\n", 111 | " 'explorer': 'https://www.omniexplorer.info/asset/31'},\n", 112 | " {'id': 'binance-coin',\n", 113 | " 'rank': 4,\n", 114 | " 'symbol': 'BNB',\n", 115 | " 'name': 'BNB',\n", 116 | " 'supply': 166801148.0,\n", 117 | " 'maxSupply': 166801148.0,\n", 118 | " 'marketCapUsd': 65157600870.97129,\n", 119 | " 'volumeUsd24Hr': 508787293.6983727,\n", 120 | " 'priceUsd': 390.63041023537374,\n", 121 | " 'changePercent24Hr': -3.3766250481883926,\n", 122 | " 'vwap24Hr': 399.59937632623144,\n", 123 | " 'explorer': 'https://etherscan.io/token/0xB8c77482e45F1F44dE1745F52C74426C631bDD52'},\n", 124 | " {'id': 'usd-coin',\n", 125 | " 'rank': 5,\n", 126 | " 'symbol': 'USDC',\n", 127 | " 'name': 'USD Coin',\n", 128 | " 'supply': 52621367085.198746,\n", 129 | " 'maxSupply': None,\n", 130 | " 'marketCapUsd': 52941186999.81836,\n", 131 | " 'volumeUsd24Hr': 955954288.1965283,\n", 132 | " 'priceUsd': 1.0060777576170112,\n", 133 | " 'changePercent24Hr': 0.444357546394629,\n", 134 | " 'vwap24Hr': 1.0018684122274557,\n", 135 | " 'explorer': 'https://etherscan.io/token/0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48'},\n", 136 | " {'id': 'xrp',\n", 137 | " 'rank': 6,\n", 138 | " 'symbol': 'XRP',\n", 139 | " 'name': 'XRP',\n", 140 | " 'supply': 45404028640.0,\n", 141 | " 'maxSupply': 100000000000.0,\n", 142 | " 'marketCapUsd': 36365362818.38755,\n", 143 | " 'volumeUsd24Hr': 1742807983.920385,\n", 144 | " 'priceUsd': 0.8009281094134107,\n", 145 | " 'changePercent24Hr': 0.8235642326801154,\n", 146 | " 'vwap24Hr': 0.8147110666919125,\n", 147 | " 'explorer': 'https://xrpcharts.ripple.com/#/graph/'},\n", 148 | " {'id': 'cardano',\n", 149 | " 'rank': 7,\n", 150 | " 'symbol': 'ADA',\n", 151 | " 'name': 'Cardano',\n", 152 | " 'supply': 33626555329.633,\n", 153 | " 'maxSupply': 45000000000.0,\n", 154 | " 'marketCapUsd': 32773905879.60214,\n", 155 | " 'volumeUsd24Hr': 528556573.71633756,\n", 156 | " 'priceUsd': 0.974643568403824,\n", 157 | " 'changePercent24Hr': -3.1328000656935377,\n", 158 | " 'vwap24Hr': 0.9955080996669287,\n", 159 | " 'explorer': 'https://cardanoexplorer.com/'},\n", 160 | " {'id': 'solana',\n", 161 | " 'rank': 8,\n", 162 | " 'symbol': 'SOL',\n", 163 | " 'name': 'Solana',\n", 164 | " 'supply': 319659506.88075066,\n", 165 | " 'maxSupply': None,\n", 166 | " 'marketCapUsd': 28319375803.00971,\n", 167 | " 'volumeUsd24Hr': 218116448.9693599,\n", 168 | " 'priceUsd': 88.59231523989769,\n", 169 | " 'changePercent24Hr': -3.206327593452971,\n", 170 | " 'vwap24Hr': 90.52580970145662,\n", 171 | " 'explorer': 'https://explorer.solana.com/'},\n", 172 | " {'id': 'avalanche',\n", 173 | " 'rank': 9,\n", 174 | " 'symbol': 'AVAX',\n", 175 | " 'name': 'Avalanche',\n", 176 | " 'supply': 245292760.96527612,\n", 177 | " 'maxSupply': None,\n", 178 | " 'marketCapUsd': 20489907862.03146,\n", 179 | " 'volumeUsd24Hr': 538392364.629816,\n", 180 | " 'priceUsd': 83.53246048272916,\n", 181 | " 'changePercent24Hr': -1.3430172783314414,\n", 182 | " 'vwap24Hr': 84.23389201657379,\n", 183 | " 'explorer': 'https://avascan.info/'},\n", 184 | " {'id': 'terra-luna',\n", 185 | " 'rank': 10,\n", 186 | " 'symbol': 'LUNA',\n", 187 | " 'name': 'Terra',\n", 188 | " 'supply': 390050079.1709837,\n", 189 | " 'maxSupply': None,\n", 190 | " 'marketCapUsd': 19375664145.92172,\n", 191 | " 'volumeUsd24Hr': 248373945.63470584,\n", 192 | " 'priceUsd': 49.67481146806314,\n", 193 | " 'changePercent24Hr': -3.9977006317367536,\n", 194 | " 'vwap24Hr': 51.13575117026866,\n", 195 | " 'explorer': 'https://finder.terra.money/'},\n", 196 | " {'id': 'polkadot',\n", 197 | " 'rank': 11,\n", 198 | " 'symbol': 'DOT',\n", 199 | " 'name': 'Polkadot',\n", 200 | " 'supply': 1085991083.27203,\n", 201 | " 'maxSupply': None,\n", 202 | " 'marketCapUsd': 18969569146.605743,\n", 203 | " 'volumeUsd24Hr': 319459296.79636973,\n", 204 | " 'priceUsd': 17.46751832386275,\n", 205 | " 'changePercent24Hr': -3.2928418099331953,\n", 206 | " 'vwap24Hr': 17.781289739080226,\n", 207 | " 'explorer': 'https://polkascan.io/polkadot'},\n", 208 | " {'id': 'dogecoin',\n", 209 | " 'rank': 12,\n", 210 | " 'symbol': 'DOGE',\n", 211 | " 'name': 'Dogecoin',\n", 212 | " 'supply': 132670764299.89409,\n", 213 | " 'maxSupply': None,\n", 214 | " 'marketCapUsd': 18458302762.680485,\n", 215 | " 'volumeUsd24Hr': 307127321.494099,\n", 216 | " 'priceUsd': 0.1391286381749986,\n", 217 | " 'changePercent24Hr': -1.8526848336568178,\n", 218 | " 'vwap24Hr': 0.1421103776351693,\n", 219 | " 'explorer': 'http://dogechain.info/chain/Dogecoin'},\n", 220 | " {'id': 'binance-usd',\n", 221 | " 'rank': 13,\n", 222 | " 'symbol': 'BUSD',\n", 223 | " 'name': 'Binance USD',\n", 224 | " 'supply': 18187576575.799255,\n", 225 | " 'maxSupply': None,\n", 226 | " 'marketCapUsd': 18265223629.17033,\n", 227 | " 'volumeUsd24Hr': 341496624.2932839,\n", 228 | " 'priceUsd': 1.0042692358186078,\n", 229 | " 'changePercent24Hr': 0.2149736785268906,\n", 230 | " 'vwap24Hr': 1.0021217161190157,\n", 231 | " 'explorer': 'https://etherscan.io/token/0x4Fabb145d64652a948d72533023f6E7A623C7C53'},\n", 232 | " {'id': 'shiba-inu',\n", 233 | " 'rank': 14,\n", 234 | " 'symbol': 'SHIB',\n", 235 | " 'name': 'Shiba Inu',\n", 236 | " 'supply': 549063278876301.94,\n", 237 | " 'maxSupply': None,\n", 238 | " 'marketCapUsd': 14530060890.11551,\n", 239 | " 'volumeUsd24Hr': 536150515.21338,\n", 240 | " 'priceUsd': 2.64633630569e-05,\n", 241 | " 'changePercent24Hr': -7.108651379655042,\n", 242 | " 'vwap24Hr': 2.76858505332e-05,\n", 243 | " 'explorer': 'https://etherscan.io/token/0x95ad61b0a150d79219dcf64e1e6cc01f0b64c4ce'},\n", 244 | " {'id': 'polygon',\n", 245 | " 'rank': 15,\n", 246 | " 'symbol': 'MATIC',\n", 247 | " 'name': 'Polygon',\n", 248 | " 'supply': 7532274364.27,\n", 249 | " 'maxSupply': 10000000000.0,\n", 250 | " 'marketCapUsd': 12115053405.841358,\n", 251 | " 'volumeUsd24Hr': 357645138.85749763,\n", 252 | " 'priceUsd': 1.6084190272343464,\n", 253 | " 'changePercent24Hr': -1.0512827325147327,\n", 254 | " 'vwap24Hr': 1.6103313961828982,\n", 255 | " 'explorer': 'https://etherscan.io/token/0x7D1AfA7B718fb893dB30A3aBc0Cfc608AaCfeBB0'},\n", 256 | " {'id': 'terrausd',\n", 257 | " 'rank': 16,\n", 258 | " 'symbol': 'UST',\n", 259 | " 'name': 'TerraUSD',\n", 260 | " 'supply': 12039285584.331615,\n", 261 | " 'maxSupply': None,\n", 262 | " 'marketCapUsd': 12084513961.078758,\n", 263 | " 'volumeUsd24Hr': 76612332.51675445,\n", 264 | " 'priceUsd': 1.0037567326092842,\n", 265 | " 'changePercent24Hr': 0.3748396192456799,\n", 266 | " 'vwap24Hr': 1.0039662946843657,\n", 267 | " 'explorer': 'https://finder.terra.money/'},\n", 268 | " {'id': 'crypto-com-coin',\n", 269 | " 'rank': 17,\n", 270 | " 'symbol': 'CRO',\n", 271 | " 'name': 'Crypto.com Coin',\n", 272 | " 'supply': 25263013692.0,\n", 273 | " 'maxSupply': 30263013692.0,\n", 274 | " 'marketCapUsd': 11090819432.243872,\n", 275 | " 'volumeUsd24Hr': 87638383.22124855,\n", 276 | " 'priceUsd': 0.4390141084298262,\n", 277 | " 'changePercent24Hr': -0.622106710909267,\n", 278 | " 'vwap24Hr': 0.4408081737580335,\n", 279 | " 'explorer': 'https://etherscan.io/token/0xa0b73e1ff0b80914ab6fe0444e65848c4c34450b'},\n", 280 | " {'id': 'wrapped-bitcoin',\n", 281 | " 'rank': 18,\n", 282 | " 'symbol': 'WBTC',\n", 283 | " 'name': 'Wrapped Bitcoin',\n", 284 | " 'supply': 262862.35686194,\n", 285 | " 'maxSupply': None,\n", 286 | " 'marketCapUsd': 10505388012.54835,\n", 287 | " 'volumeUsd24Hr': 88738103.27856533,\n", 288 | " 'priceUsd': 39965.357299394396,\n", 289 | " 'changePercent24Hr': -0.5795176796849094,\n", 290 | " 'vwap24Hr': 40094.005746827,\n", 291 | " 'explorer': 'https://etherscan.io/token/0x2260fac5e5542a773aa44fbcfedf7c193bc2c599'},\n", 292 | " {'id': 'multi-collateral-dai',\n", 293 | " 'rank': 19,\n", 294 | " 'symbol': 'DAI',\n", 295 | " 'name': 'Multi Collateral DAI',\n", 296 | " 'supply': 10210440319.68423,\n", 297 | " 'maxSupply': None,\n", 298 | " 'marketCapUsd': 10261836686.78883,\n", 299 | " 'volumeUsd24Hr': 57148645.79258416,\n", 300 | " 'priceUsd': 1.0050337072149098,\n", 301 | " 'changePercent24Hr': 0.3054670743947476,\n", 302 | " 'vwap24Hr': 1.000917133576841,\n", 303 | " 'explorer': 'https://etherscan.io/token/0x89d24a6b4ccb1b6faa2625fe562bdd9a23260359'},\n", 304 | " {'id': 'litecoin',\n", 305 | " 'rank': 20,\n", 306 | " 'symbol': 'LTC',\n", 307 | " 'name': 'Litecoin',\n", 308 | " 'supply': 69677619.06850561,\n", 309 | " 'maxSupply': 84000000.0,\n", 310 | " 'marketCapUsd': 7949112516.865393,\n", 311 | " 'volumeUsd24Hr': 320215916.00606644,\n", 312 | " 'priceUsd': 114.0841582007845,\n", 313 | " 'changePercent24Hr': -2.183256705450328,\n", 314 | " 'vwap24Hr': 115.48590638260806,\n", 315 | " 'explorer': 'http://explorer.litecoin.net/chain/Litecoin'},\n", 316 | " {'id': 'chainlink',\n", 317 | " 'rank': 21,\n", 318 | " 'symbol': 'LINK',\n", 319 | " 'name': 'Chainlink',\n", 320 | " 'supply': 467009549.5205637,\n", 321 | " 'maxSupply': 1000000000.0,\n", 322 | " 'marketCapUsd': 6986582664.015613,\n", 323 | " 'volumeUsd24Hr': 257122096.9556117,\n", 324 | " 'priceUsd': 14.960256532630012,\n", 325 | " 'changePercent24Hr': -2.952709424248067,\n", 326 | " 'vwap24Hr': 15.314529059047992,\n", 327 | " 'explorer': 'https://etherscan.io/token/0x514910771af9ca656af840dff83e8264ecf986ca'},\n", 328 | " {'id': 'cosmos',\n", 329 | " 'rank': 22,\n", 330 | " 'symbol': 'ATOM',\n", 331 | " 'name': 'Cosmos',\n", 332 | " 'supply': 248453201.0,\n", 333 | " 'maxSupply': None,\n", 334 | " 'marketCapUsd': 6459198489.777206,\n", 335 | " 'volumeUsd24Hr': 225192639.22649837,\n", 336 | " 'priceUsd': 25.997646493502838,\n", 337 | " 'changePercent24Hr': -4.636651991494304,\n", 338 | " 'vwap24Hr': 26.68747808270693,\n", 339 | " 'explorer': 'https://www.mintscan.io/'},\n", 340 | " {'id': 'tron',\n", 341 | " 'rank': 23,\n", 342 | " 'symbol': 'TRX',\n", 343 | " 'name': 'TRON',\n", 344 | " 'supply': 101771733720.91115,\n", 345 | " 'maxSupply': None,\n", 346 | " 'marketCapUsd': 6392944759.898668,\n", 347 | " 'volumeUsd24Hr': 366641973.91556555,\n", 348 | " 'priceUsd': 0.0628165063732731,\n", 349 | " 'changePercent24Hr': -1.6395725646271886,\n", 350 | " 'vwap24Hr': 0.06384960970898,\n", 351 | " 'explorer': 'https://tronscan.org/#/'},\n", 352 | " {'id': 'near-protocol',\n", 353 | " 'rank': 24,\n", 354 | " 'symbol': 'NEAR',\n", 355 | " 'name': 'NEAR Protocol',\n", 356 | " 'supply': 637607105.0,\n", 357 | " 'maxSupply': 1000000000.0,\n", 358 | " 'marketCapUsd': 6287563794.327358,\n", 359 | " 'volumeUsd24Hr': 128820486.24622315,\n", 360 | " 'priceUsd': 9.861188410576696,\n", 361 | " 'changePercent24Hr': -4.827639756736576,\n", 362 | " 'vwap24Hr': 10.10875441091875,\n", 363 | " 'explorer': 'https://explorer.nearprotocol.com/'},\n", 364 | " {'id': 'uniswap',\n", 365 | " 'rank': 25,\n", 366 | " 'symbol': 'UNI',\n", 367 | " 'name': 'Uniswap',\n", 368 | " 'supply': 632544237.5965742,\n", 369 | " 'maxSupply': 1000000000.0,\n", 370 | " 'marketCapUsd': 6200677979.49193,\n", 371 | " 'volumeUsd24Hr': 100570209.68021473,\n", 372 | " 'priceUsd': 9.802757832483197,\n", 373 | " 'changePercent24Hr': -5.1464871182182375,\n", 374 | " 'vwap24Hr': 10.046666948793652,\n", 375 | " 'explorer': 'https://etherscan.io/token/0x1f9840a85d5af5bf1d1762f925bdaddc4201f984'},\n", 376 | " {'id': 'bitcoin-cash',\n", 377 | " 'rank': 26,\n", 378 | " 'symbol': 'BCH',\n", 379 | " 'name': 'Bitcoin Cash',\n", 380 | " 'supply': 18988031.25,\n", 381 | " 'maxSupply': 21000000.0,\n", 382 | " 'marketCapUsd': 5873256252.165868,\n", 383 | " 'volumeUsd24Hr': 123039909.00677446,\n", 384 | " 'priceUsd': 309.31359733073265,\n", 385 | " 'changePercent24Hr': -1.3903440131016958,\n", 386 | " 'vwap24Hr': 311.8125647873954,\n", 387 | " 'explorer': 'https://blockchair.com/bitcoin-cash/blocks'},\n", 388 | " {'id': 'ftx-token',\n", 389 | " 'rank': 27,\n", 390 | " 'symbol': 'FTT',\n", 391 | " 'name': 'FTX Token',\n", 392 | " 'supply': 137926022.11874375,\n", 393 | " 'maxSupply': 352170015.0,\n", 394 | " 'marketCapUsd': 5829026850.107838,\n", 395 | " 'volumeUsd24Hr': 20609613.151582118,\n", 396 | " 'priceUsd': 42.2619804483993,\n", 397 | " 'changePercent24Hr': -2.723034602919218,\n", 398 | " 'vwap24Hr': 42.74308848457945,\n", 399 | " 'explorer': 'https://etherscan.io/token/0x50d1c9771902476076ecfc8b2a83ad6b9355a4c9'},\n", 400 | " {'id': 'algorand',\n", 401 | " 'rank': 28,\n", 402 | " 'symbol': 'ALGO',\n", 403 | " 'name': 'Algorand',\n", 404 | " 'supply': 6616534154.919784,\n", 405 | " 'maxSupply': 10000000000.0,\n", 406 | " 'marketCapUsd': 5796195943.847043,\n", 407 | " 'volumeUsd24Hr': 75140997.17120561,\n", 408 | " 'priceUsd': 0.8760169309391729,\n", 409 | " 'changePercent24Hr': -3.9470598018564362,\n", 410 | " 'vwap24Hr': 0.893868719350993,\n", 411 | " 'explorer': 'https://algoexplorer.io/'},\n", 412 | " {'id': 'unus-sed-leo',\n", 413 | " 'rank': 29,\n", 414 | " 'symbol': 'LEO',\n", 415 | " 'name': 'UNUS SED LEO',\n", 416 | " 'supply': 953954130.0,\n", 417 | " 'maxSupply': None,\n", 418 | " 'marketCapUsd': 5533744249.984172,\n", 419 | " 'volumeUsd24Hr': 5080976.247165999,\n", 420 | " 'priceUsd': 5.800849407700737,\n", 421 | " 'changePercent24Hr': 2.603069849974849,\n", 422 | " 'vwap24Hr': 5.731754855912861,\n", 423 | " 'explorer': 'https://eospark.com/account/bitfinexleo1'},\n", 424 | " {'id': 'decentraland',\n", 425 | " 'rank': 30,\n", 426 | " 'symbol': 'MANA',\n", 427 | " 'name': 'Decentraland',\n", 428 | " 'supply': 1832126198.3634298,\n", 429 | " 'maxSupply': None,\n", 430 | " 'marketCapUsd': 5115181194.128434,\n", 431 | " 'volumeUsd24Hr': 208808326.92277884,\n", 432 | " 'priceUsd': 2.791937148596879,\n", 433 | " 'changePercent24Hr': -6.348191746281253,\n", 434 | " 'vwap24Hr': 2.889318938066877,\n", 435 | " 'explorer': 'https://etherscan.io/token/decentraland'},\n", 436 | " {'id': 'steth',\n", 437 | " 'rank': 31,\n", 438 | " 'symbol': 'STETH',\n", 439 | " 'name': 'Lido stETH',\n", 440 | " 'supply': 1901134.22832698,\n", 441 | " 'maxSupply': 141554.0,\n", 442 | " 'marketCapUsd': 5112801078.2006235,\n", 443 | " 'volumeUsd24Hr': 124540.08142290822,\n", 444 | " 'priceUsd': 2689.342499871746,\n", 445 | " 'changePercent24Hr': -3.390174075592614,\n", 446 | " 'vwap24Hr': 2733.038428004193,\n", 447 | " 'explorer': 'https://etherscan.io/token/0xae7ab96520de3a18e5e111b5eaab095312d7fe84'},\n", 448 | " {'id': 'stellar',\n", 449 | " 'rank': 32,\n", 450 | " 'symbol': 'XLM',\n", 451 | " 'name': 'Stellar',\n", 452 | " 'supply': 24927502934.036396,\n", 453 | " 'maxSupply': 50001806812.0,\n", 454 | " 'marketCapUsd': 4986891716.671906,\n", 455 | " 'volumeUsd24Hr': 184717245.43866718,\n", 456 | " 'priceUsd': 0.2000558070284179,\n", 457 | " 'changePercent24Hr': -1.8500664692454167,\n", 458 | " 'vwap24Hr': 0.204254697479701,\n", 459 | " 'explorer': 'https://dashboard.stellar.org/'},\n", 460 | " {'id': 'fantom',\n", 461 | " 'rank': 33,\n", 462 | " 'symbol': 'FTM',\n", 463 | " 'name': 'Fantom',\n", 464 | " 'supply': 2545006273.0,\n", 465 | " 'maxSupply': 3175000000.0,\n", 466 | " 'marketCapUsd': 4671869109.912609,\n", 467 | " 'volumeUsd24Hr': 190420532.76678497,\n", 468 | " 'priceUsd': 1.8357004300840123,\n", 469 | " 'changePercent24Hr': -3.0237308962672946,\n", 470 | " 'vwap24Hr': 1.8638073653263225,\n", 471 | " 'explorer': 'https://etherscan.io/token/0x4e15361fd6b4bb609fa63c81a2be19d873717870'},\n", 472 | " {'id': 'bitcoin-bep2',\n", 473 | " 'rank': 34,\n", 474 | " 'symbol': 'BTCB',\n", 475 | " 'name': 'Bitcoin BEP2',\n", 476 | " 'supply': 105148.14224812,\n", 477 | " 'maxSupply': None,\n", 478 | " 'marketCapUsd': 4181868460.3704367,\n", 479 | " 'volumeUsd24Hr': 227850207.99911842,\n", 480 | " 'priceUsd': 39771.2063281385,\n", 481 | " 'changePercent24Hr': -0.9054048986788773,\n", 482 | " 'vwap24Hr': 40010.16810447089,\n", 483 | " 'explorer': 'https://explorer.binance.org/asset/BTCB-1DE'},\n", 484 | " {'id': 'internet-computer',\n", 485 | " 'rank': 35,\n", 486 | " 'symbol': 'ICP',\n", 487 | " 'name': 'Internet Computer',\n", 488 | " 'supply': 206874282.86447436,\n", 489 | " 'maxSupply': None,\n", 490 | " 'marketCapUsd': 4056467560.7743583,\n", 491 | " 'volumeUsd24Hr': 131511942.83318688,\n", 492 | " 'priceUsd': 19.608370381309285,\n", 493 | " 'changePercent24Hr': -1.5775186930441052,\n", 494 | " 'vwap24Hr': 19.820665917431395,\n", 495 | " 'explorer': 'https://www.dfinityexplorer.org/#/'},\n", 496 | " {'id': 'ethereum-classic',\n", 497 | " 'rank': 36,\n", 498 | " 'symbol': 'ETC',\n", 499 | " 'name': 'Ethereum Classic',\n", 500 | " 'supply': 133083338.91222619,\n", 501 | " 'maxSupply': 210700000.0,\n", 502 | " 'marketCapUsd': 3764632049.270399,\n", 503 | " 'volumeUsd24Hr': 134147818.41026556,\n", 504 | " 'priceUsd': 28.287778771115182,\n", 505 | " 'changePercent24Hr': -2.219847068657417,\n", 506 | " 'vwap24Hr': 28.515223919660052,\n", 507 | " 'explorer': 'http://gastracker.io/'},\n", 508 | " {'id': 'the-sandbox',\n", 509 | " 'rank': 37,\n", 510 | " 'symbol': 'SAND',\n", 511 | " 'name': 'The Sandbox',\n", 512 | " 'supply': 1088944772.2233226,\n", 513 | " 'maxSupply': 3000000000.0,\n", 514 | " 'marketCapUsd': 3695041686.141474,\n", 515 | " 'volumeUsd24Hr': 269745471.4781308,\n", 516 | " 'priceUsd': 3.3932314846392306,\n", 517 | " 'changePercent24Hr': -5.5579035768564005,\n", 518 | " 'vwap24Hr': 3.5036708646009034,\n", 519 | " 'explorer': 'https://etherscan.io/token/0x3845badAde8e6dFF049820680d1F14bD3903a5d0'},\n", 520 | " {'id': 'elrond-egld',\n", 521 | " 'rank': 38,\n", 522 | " 'symbol': 'EGLD',\n", 523 | " 'name': 'Elrond',\n", 524 | " 'supply': 21536024.21504843,\n", 525 | " 'maxSupply': 31415926.0,\n", 526 | " 'marketCapUsd': 3444917759.294095,\n", 527 | " 'volumeUsd24Hr': 73777284.59411168,\n", 528 | " 'priceUsd': 159.96071163808116,\n", 529 | " 'changePercent24Hr': -2.8129060952430054,\n", 530 | " 'vwap24Hr': 162.850128330527,\n", 531 | " 'explorer': 'https://explorer.elrond.com/'},\n", 532 | " {'id': 'hedera-hashgraph',\n", 533 | " 'rank': 39,\n", 534 | " 'symbol': 'HBAR',\n", 535 | " 'name': 'Hedera Hashgraph',\n", 536 | " 'supply': 14832756028.0,\n", 537 | " 'maxSupply': 50000000000.0,\n", 538 | " 'marketCapUsd': 3410694975.9385676,\n", 539 | " 'volumeUsd24Hr': 33486764.075605024,\n", 540 | " 'priceUsd': 0.229943442034653,\n", 541 | " 'changePercent24Hr': -0.4473451254408215,\n", 542 | " 'vwap24Hr': 0.2302444615000423,\n", 543 | " 'explorer': 'https://hash-hash.info/'},\n", 544 | " {'id': 'filecoin',\n", 545 | " 'rank': 40,\n", 546 | " 'symbol': 'FIL',\n", 547 | " 'name': 'Filecoin',\n", 548 | " 'supply': 165594480.0,\n", 549 | " 'maxSupply': None,\n", 550 | " 'marketCapUsd': 3384441575.2035108,\n", 551 | " 'volumeUsd24Hr': 195344477.5838861,\n", 552 | " 'priceUsd': 20.438130396638286,\n", 553 | " 'changePercent24Hr': -2.935870516238413,\n", 554 | " 'vwap24Hr': 20.70435354022569,\n", 555 | " 'explorer': 'https://protocol.ai'},\n", 556 | " {'id': 'klaytn',\n", 557 | " 'rank': 41,\n", 558 | " 'symbol': 'KLAY',\n", 559 | " 'name': 'Klaytn',\n", 560 | " 'supply': 2634113550.6,\n", 561 | " 'maxSupply': None,\n", 562 | " 'marketCapUsd': 3381879144.63109,\n", 563 | " 'volumeUsd24Hr': 47674086.76083397,\n", 564 | " 'priceUsd': 1.2838775093278585,\n", 565 | " 'changePercent24Hr': 0.1331030538461247,\n", 566 | " 'vwap24Hr': 1.2865239283677252,\n", 567 | " 'explorer': 'https://scope.klaytn.com/blocks'},\n", 568 | " {'id': 'vechain',\n", 569 | " 'rank': 42,\n", 570 | " 'symbol': 'VET',\n", 571 | " 'name': 'VeChain',\n", 572 | " 'supply': 64315576989.0,\n", 573 | " 'maxSupply': 86712634466.0,\n", 574 | " 'marketCapUsd': 3340017182.764552,\n", 575 | " 'volumeUsd24Hr': 99726336.59189297,\n", 576 | " 'priceUsd': 0.0519316989620695,\n", 577 | " 'changePercent24Hr': -3.8911063418157896,\n", 578 | " 'vwap24Hr': 0.0528440569508168,\n", 579 | " 'explorer': 'https://explore.veforge.com/'},\n", 580 | " {'id': 'axie-infinity',\n", 581 | " 'rank': 43,\n", 582 | " 'symbol': 'AXS',\n", 583 | " 'name': 'Axie Infinity',\n", 584 | " 'supply': 60907500.0,\n", 585 | " 'maxSupply': 270000000.0,\n", 586 | " 'marketCapUsd': 3218393787.4853306,\n", 587 | " 'volumeUsd24Hr': 84699167.67215706,\n", 588 | " 'priceUsd': 52.84068115561024,\n", 589 | " 'changePercent24Hr': -5.880465494802879,\n", 590 | " 'vwap24Hr': 54.96887824889594,\n", 591 | " 'explorer': 'https://etherscan.io/token/0xf5d669627376ebd411e34b98f19c868c8aba5ada'},\n", 592 | " {'id': 'tezos',\n", 593 | " 'rank': 44,\n", 594 | " 'symbol': 'XTZ',\n", 595 | " 'name': 'Tezos',\n", 596 | " 'supply': 878664618.165657,\n", 597 | " 'maxSupply': None,\n", 598 | " 'marketCapUsd': 3105736236.2469106,\n", 599 | " 'volumeUsd24Hr': 51157423.54364294,\n", 600 | " 'priceUsd': 3.53460941983825,\n", 601 | " 'changePercent24Hr': -3.7577359184260377,\n", 602 | " 'vwap24Hr': 3.597883232056851,\n", 603 | " 'explorer': 'https://tzkt.io/'},\n", 604 | " {'id': 'theta',\n", 605 | " 'rank': 45,\n", 606 | " 'symbol': 'THETA',\n", 607 | " 'name': 'THETA',\n", 608 | " 'supply': 1000000000.0,\n", 609 | " 'maxSupply': 1000000000.0,\n", 610 | " 'marketCapUsd': 3104064879.0144362,\n", 611 | " 'volumeUsd24Hr': 127335508.6461018,\n", 612 | " 'priceUsd': 3.104064879014436,\n", 613 | " 'changePercent24Hr': -6.509975761018726,\n", 614 | " 'vwap24Hr': 3.2192822680539868,\n", 615 | " 'explorer': 'https://explorer.thetatoken.org/'},\n", 616 | " {'id': 'monero',\n", 617 | " 'rank': 46,\n", 618 | " 'symbol': 'XMR',\n", 619 | " 'name': 'Monero',\n", 620 | " 'supply': 18081498.20976178,\n", 621 | " 'maxSupply': None,\n", 622 | " 'marketCapUsd': 2874142697.8018017,\n", 623 | " 'volumeUsd24Hr': 102262261.69370995,\n", 624 | " 'priceUsd': 158.95489767823105,\n", 625 | " 'changePercent24Hr': -1.4070676465621692,\n", 626 | " 'vwap24Hr': 162.26974553688655,\n", 627 | " 'explorer': 'http://moneroblocks.info/'},\n", 628 | " {'id': 'bittorrent',\n", 629 | " 'rank': 47,\n", 630 | " 'symbol': 'BTTOLD',\n", 631 | " 'name': 'BitTorrent',\n", 632 | " 'supply': 990000000000.0,\n", 633 | " 'maxSupply': None,\n", 634 | " 'marketCapUsd': 2744987384.443788,\n", 635 | " 'volumeUsd24Hr': 243293006.22397557,\n", 636 | " 'priceUsd': 0.0027727145297412,\n", 637 | " 'changePercent24Hr': 0.1241952491822106,\n", 638 | " 'vwap24Hr': 0.002194070077291,\n", 639 | " 'explorer': 'https://tronscan.org/#/token/1002000'},\n", 640 | " {'id': 'frax',\n", 641 | " 'rank': 48,\n", 642 | " 'symbol': 'FRAX',\n", 643 | " 'name': 'Frax',\n", 644 | " 'supply': 2653242158.006508,\n", 645 | " 'maxSupply': None,\n", 646 | " 'marketCapUsd': 2653928757.451277,\n", 647 | " 'volumeUsd24Hr': 1028889.4780571252,\n", 648 | " 'priceUsd': 1.0002587775272216,\n", 649 | " 'changePercent24Hr': -0.087083722221097,\n", 650 | " 'vwap24Hr': 1.001700528540497,\n", 651 | " 'explorer': 'https://etherscan.io/token/0x853d955acef822db058eb8505911ed77f175b99e'},\n", 652 | " {'id': 'helium',\n", 653 | " 'rank': 49,\n", 654 | " 'symbol': 'HNT',\n", 655 | " 'name': 'Helium',\n", 656 | " 'supply': 111051702.15570812,\n", 657 | " 'maxSupply': 223000000.0,\n", 658 | " 'marketCapUsd': 2582649622.6269836,\n", 659 | " 'volumeUsd24Hr': 13405625.815261424,\n", 660 | " 'priceUsd': 23.25628128604271,\n", 661 | " 'changePercent24Hr': -7.9316975454277525,\n", 662 | " 'vwap24Hr': 24.349034793929118,\n", 663 | " 'explorer': 'https://explorer.helium.com/'},\n", 664 | " {'id': 'iota',\n", 665 | " 'rank': 50,\n", 666 | " 'symbol': 'MIOTA',\n", 667 | " 'name': 'IOTA',\n", 668 | " 'supply': 2779530283.0,\n", 669 | " 'maxSupply': 2779530283.0,\n", 670 | " 'marketCapUsd': 2283094518.3827133,\n", 671 | " 'volumeUsd24Hr': 20933958.117726214,\n", 672 | " 'priceUsd': 0.8213958064592575,\n", 673 | " 'changePercent24Hr': 1.4680720093085355,\n", 674 | " 'vwap24Hr': 0.8181868785517682,\n", 675 | " 'explorer': 'https://thetangle.org/'},\n", 676 | " {'id': 'eos',\n", 677 | " 'rank': 51,\n", 678 | " 'symbol': 'EOS',\n", 679 | " 'name': 'EOS',\n", 680 | " 'supply': 980948936.2094,\n", 681 | " 'maxSupply': None,\n", 682 | " 'marketCapUsd': 2220315435.007206,\n", 683 | " 'volumeUsd24Hr': 143423606.97487482,\n", 684 | " 'priceUsd': 2.26343630442884,\n", 685 | " 'changePercent24Hr': -2.3715645009577524,\n", 686 | " 'vwap24Hr': 2.2886715907630175,\n", 687 | " 'explorer': 'https://bloks.io/'},\n", 688 | " {'id': 'pancakeswap',\n", 689 | " 'rank': 52,\n", 690 | " 'symbol': 'CAKE',\n", 691 | " 'name': 'PancakeSwap',\n", 692 | " 'supply': 269775795.77349913,\n", 693 | " 'maxSupply': None,\n", 694 | " 'marketCapUsd': 2053667044.140363,\n", 695 | " 'volumeUsd24Hr': 27304141.43208945,\n", 696 | " 'priceUsd': 7.612495547467868,\n", 697 | " 'changePercent24Hr': -2.762293875672329,\n", 698 | " 'vwap24Hr': 7.6924709877900215,\n", 699 | " 'explorer': 'https://bscscan.com/token/0x0e09fabb73bd3ade0a17ecc321fd13a19e81ce82'},\n", 700 | " {'id': 'flow',\n", 701 | " 'rank': 53,\n", 702 | " 'symbol': 'FLOW',\n", 703 | " 'name': 'Flow',\n", 704 | " 'supply': 329356498.0,\n", 705 | " 'maxSupply': None,\n", 706 | " 'marketCapUsd': 1988610294.2634284,\n", 707 | " 'volumeUsd24Hr': 43865892.11248289,\n", 708 | " 'priceUsd': 6.037865675458537,\n", 709 | " 'changePercent24Hr': -4.972008941064697,\n", 710 | " 'vwap24Hr': 6.185684566359474,\n", 711 | " 'explorer': 'https://flowscan.org/'},\n", 712 | " {'id': 'trustnote',\n", 713 | " 'rank': 54,\n", 714 | " 'symbol': 'TTT',\n", 715 | " 'name': 'TrustNote',\n", 716 | " 'supply': 309999945.003528,\n", 717 | " 'maxSupply': None,\n", 718 | " 'marketCapUsd': 1925591165.1607,\n", 719 | " 'volumeUsd24Hr': 389964.3020877368,\n", 720 | " 'priceUsd': 6.211585505728995,\n", 721 | " 'changePercent24Hr': -1.8225943118824646,\n", 722 | " 'vwap24Hr': 6.350107772327622,\n", 723 | " 'explorer': 'https://explorer.trustnote.org/'},\n", 724 | " {'id': 'aave',\n", 725 | " 'rank': 55,\n", 726 | " 'symbol': 'AAVE',\n", 727 | " 'name': 'Aave',\n", 728 | " 'supply': 13522530.56780633,\n", 729 | " 'maxSupply': 16000000.0,\n", 730 | " 'marketCapUsd': 1924387186.154958,\n", 731 | " 'volumeUsd24Hr': 45285062.51266997,\n", 732 | " 'priceUsd': 142.30969392196675,\n", 733 | " 'changePercent24Hr': -3.9638414532120296,\n", 734 | " 'vwap24Hr': 145.39563986195364,\n", 735 | " 'explorer': 'https://etherscan.io/token/0x80fB784B7eD66730e8b1DBd9820aFD29931aab03'},\n", 736 | " {'id': 'the-graph',\n", 737 | " 'rank': 56,\n", 738 | " 'symbol': 'GRT',\n", 739 | " 'name': 'The Graph',\n", 740 | " 'supply': 4715735200.0,\n", 741 | " 'maxSupply': 10057044431.0,\n", 742 | " 'marketCapUsd': 1913488308.6364892,\n", 743 | " 'volumeUsd24Hr': 42448969.65308949,\n", 744 | " 'priceUsd': 0.4057666996731473,\n", 745 | " 'changePercent24Hr': -1.7591337254583905,\n", 746 | " 'vwap24Hr': 0.4135958425251153,\n", 747 | " 'explorer': 'https://etherscan.io/token/0xc944e90c64b2c07662a292be6244bdf05cda44a7'},\n", 748 | " {'id': 'harmony',\n", 749 | " 'rank': 57,\n", 750 | " 'symbol': 'ONE',\n", 751 | " 'name': 'Harmony',\n", 752 | " 'supply': 11712496278.100666,\n", 753 | " 'maxSupply': None,\n", 754 | " 'marketCapUsd': 1882972968.946115,\n", 755 | " 'volumeUsd24Hr': 55572289.31771111,\n", 756 | " 'priceUsd': 0.160766153024679,\n", 757 | " 'changePercent24Hr': -5.148229207086469,\n", 758 | " 'vwap24Hr': 0.1662645753736242,\n", 759 | " 'explorer': 'https://explorer.harmony.one'},\n", 760 | " {'id': 'gala',\n", 761 | " 'rank': 58,\n", 762 | " 'symbol': 'GALA',\n", 763 | " 'name': 'Gala',\n", 764 | " 'supply': 6977205436.0,\n", 765 | " 'maxSupply': None,\n", 766 | " 'marketCapUsd': 1879594631.0779738,\n", 767 | " 'volumeUsd24Hr': 439298096.48144215,\n", 768 | " 'priceUsd': 0.2693907536934353,\n", 769 | " 'changePercent24Hr': -8.19696655752276,\n", 770 | " 'vwap24Hr': 0.2830255338439953,\n", 771 | " 'explorer': 'https://ethplorer.io/es/address/0x15d4c048f83bd7e37d49ea4c83a07267ec4203da#chart=candlestick'},\n", 772 | " {'id': 'maker',\n", 773 | " 'rank': 59,\n", 774 | " 'symbol': 'MKR',\n", 775 | " 'name': 'Maker',\n", 776 | " 'supply': 977631.03695089,\n", 777 | " 'maxSupply': 1005577.0,\n", 778 | " 'marketCapUsd': 1827498331.4295301,\n", 779 | " 'volumeUsd24Hr': 10780686.306705244,\n", 780 | " 'priceUsd': 1869.312923134346,\n", 781 | " 'changePercent24Hr': -2.4116019837930596,\n", 782 | " 'vwap24Hr': 1893.8685118562748,\n", 783 | " 'explorer': 'https://etherscan.io/token/Maker'},\n", 784 | " {'id': 'neo',\n", 785 | " 'rank': 60,\n", 786 | " 'symbol': 'NEO',\n", 787 | " 'name': 'Neo',\n", 788 | " 'supply': 70538831.0,\n", 789 | " 'maxSupply': 100000000.0,\n", 790 | " 'marketCapUsd': 1817724875.3010962,\n", 791 | " 'volumeUsd24Hr': 156380102.66199222,\n", 792 | " 'priceUsd': 25.769138069513744,\n", 793 | " 'changePercent24Hr': -3.3038014105601974,\n", 794 | " 'vwap24Hr': 25.59842350240246,\n", 795 | " 'explorer': 'https://neotracker.io'},\n", 796 | " {'id': 'stacks',\n", 797 | " 'rank': 61,\n", 798 | " 'symbol': 'STX',\n", 799 | " 'name': 'Stacks',\n", 800 | " 'supply': 1299051560.415593,\n", 801 | " 'maxSupply': 1818000000.0,\n", 802 | " 'marketCapUsd': 1685466768.9567587,\n", 803 | " 'volumeUsd24Hr': 2308256.1305777356,\n", 804 | " 'priceUsd': 1.2974594853013715,\n", 805 | " 'changePercent24Hr': -2.5877001324138513,\n", 806 | " 'vwap24Hr': 1.3104955782313301,\n", 807 | " 'explorer': 'https://explorer.xinfin.network/'},\n", 808 | " {'id': 'bitcoin-sv',\n", 809 | " 'rank': 62,\n", 810 | " 'symbol': 'BSV',\n", 811 | " 'name': 'Bitcoin SV',\n", 812 | " 'supply': 18984107.89423905,\n", 813 | " 'maxSupply': 21000000.0,\n", 814 | " 'marketCapUsd': 1646236044.040442,\n", 815 | " 'volumeUsd24Hr': 21795030.93397718,\n", 816 | " 'priceUsd': 86.71653433554344,\n", 817 | " 'changePercent24Hr': -1.5850607896807465,\n", 818 | " 'vwap24Hr': 87.4345093178796,\n", 819 | " 'explorer': 'https://bsvexplorer.io/'},\n", 820 | " {'id': 'kucoin-token',\n", 821 | " 'rank': 63,\n", 822 | " 'symbol': 'KCS',\n", 823 | " 'name': 'KuCoin Token',\n", 824 | " 'supply': 80118638.0,\n", 825 | " 'maxSupply': 170118638.0,\n", 826 | " 'marketCapUsd': 1584587978.3815525,\n", 827 | " 'volumeUsd24Hr': 7462640.67718037,\n", 828 | " 'priceUsd': 19.77801942141793,\n", 829 | " 'changePercent24Hr': -0.4833248337331285,\n", 830 | " 'vwap24Hr': 19.755323911949787,\n", 831 | " 'explorer': 'https://etherscan.io/token/0xf34960d9d60be18cc1d5afc1a6f012a723a28811'},\n", 832 | " {'id': 'huobi-token',\n", 833 | " 'rank': 64,\n", 834 | " 'symbol': 'HT',\n", 835 | " 'name': 'Huobi Token',\n", 836 | " 'supply': 156124154.25441208,\n", 837 | " 'maxSupply': 500000000.0,\n", 838 | " 'marketCapUsd': 1529769807.3869238,\n", 839 | " 'volumeUsd24Hr': 33807938.614081606,\n", 840 | " 'priceUsd': 9.798418538710466,\n", 841 | " 'changePercent24Hr': 1.945992540301258,\n", 842 | " 'vwap24Hr': 9.663203033417844,\n", 843 | " 'explorer': 'https://etherscan.io/token/0x6f259637dcd74c767781e37bc6133cd6a68aa161'},\n", 844 | " {'id': 'zcash',\n", 845 | " 'rank': 65,\n", 846 | " 'symbol': 'ZEC',\n", 847 | " 'name': 'Zcash',\n", 848 | " 'supply': 13783531.25,\n", 849 | " 'maxSupply': 21000000.0,\n", 850 | " 'marketCapUsd': 1517472435.4759276,\n", 851 | " 'volumeUsd24Hr': 149745549.00104344,\n", 852 | " 'priceUsd': 110.09315450102292,\n", 853 | " 'changePercent24Hr': 0.2315963535854102,\n", 854 | " 'vwap24Hr': 111.54114135662637,\n", 855 | " 'explorer': 'https://explorer.zcha.in/'},\n", 856 | " {'id': 'ecash',\n", 857 | " 'rank': 66,\n", 858 | " 'symbol': 'XEC',\n", 859 | " 'name': 'eCash',\n", 860 | " 'supply': 18983998423313.0,\n", 861 | " 'maxSupply': 21000000000000.0,\n", 862 | " 'marketCapUsd': 1450791410.3142405,\n", 863 | " 'volumeUsd24Hr': 4665451.9437637925,\n", 864 | " 'priceUsd': 7.64218041934e-05,\n", 865 | " 'changePercent24Hr': -3.7426896353518266,\n", 866 | " 'vwap24Hr': 7.78275209321e-05,\n", 867 | " 'explorer': 'https://explorer.bitcoinabc.org/'},\n", 868 | " {'id': 'trueusd',\n", 869 | " 'rank': 67,\n", 870 | " 'symbol': 'TUSD',\n", 871 | " 'name': 'TrueUSD',\n", 872 | " 'supply': 1428419071.077317,\n", 873 | " 'maxSupply': None,\n", 874 | " 'marketCapUsd': 1434399881.35718,\n", 875 | " 'volumeUsd24Hr': 65230710.462881334,\n", 876 | " 'priceUsd': 1.0041870137419493,\n", 877 | " 'changePercent24Hr': 0.2319430158781222,\n", 878 | " 'vwap24Hr': 1.002362930967088,\n", 879 | " 'explorer': 'https://etherscan.io/token/0x8dd5fbce2f6a956c3022ba3663759011dd51e73e'},\n", 880 | " {'id': 'enjin-coin',\n", 881 | " 'rank': 68,\n", 882 | " 'symbol': 'ENJ',\n", 883 | " 'name': 'Enjin Coin',\n", 884 | " 'supply': 860212705.567558,\n", 885 | " 'maxSupply': 1000000000.0,\n", 886 | " 'marketCapUsd': 1399718368.8445787,\n", 887 | " 'volumeUsd24Hr': 54780341.71216046,\n", 888 | " 'priceUsd': 1.6271770456134584,\n", 889 | " 'changePercent24Hr': -3.330243607032236,\n", 890 | " 'vwap24Hr': 1.6591288597916694,\n", 891 | " 'explorer': 'https://etherscan.io/token/0xf629cbd94d3791c9250152bd8dfbdf380e2a3b9c'},\n", 892 | " {'id': 'quant',\n", 893 | " 'rank': 69,\n", 894 | " 'symbol': 'QNT',\n", 895 | " 'name': 'Quant',\n", 896 | " 'supply': 12072738.0,\n", 897 | " 'maxSupply': 14612493.0,\n", 898 | " 'marketCapUsd': 1313285297.6671798,\n", 899 | " 'volumeUsd24Hr': 23460469.441209484,\n", 900 | " 'priceUsd': 108.78106504648571,\n", 901 | " 'changePercent24Hr': -4.436150987868466,\n", 902 | " 'vwap24Hr': 110.79112409565649,\n", 903 | " 'explorer': 'https://etherscan.io/token/0x4a220e6096b25eadb88358cb44068a3248254675'},\n", 904 | " {'id': 'thorchain',\n", 905 | " 'rank': 70,\n", 906 | " 'symbol': 'RUNE',\n", 907 | " 'name': 'THORChain',\n", 908 | " 'supply': 330688061.33445597,\n", 909 | " 'maxSupply': 500000000.0,\n", 910 | " 'marketCapUsd': 1296227679.2011926,\n", 911 | " 'volumeUsd24Hr': 18990686.342680104,\n", 912 | " 'priceUsd': 3.91978976794749,\n", 913 | " 'changePercent24Hr': -5.4710500198540695,\n", 914 | " 'vwap24Hr': 4.017713897104055,\n", 915 | " 'explorer': 'https://explorer.binance.org/asset/RUNE-B1A'},\n", 916 | " {'id': 'convex-finance',\n", 917 | " 'rank': 71,\n", 918 | " 'symbol': 'CVX',\n", 919 | " 'name': 'Convex Finance',\n", 920 | " 'supply': 51132251.89124198,\n", 921 | " 'maxSupply': 100000000.0,\n", 922 | " 'marketCapUsd': 1281571640.2071064,\n", 923 | " 'volumeUsd24Hr': 1328263.9373596336,\n", 924 | " 'priceUsd': 25.06386072987754,\n", 925 | " 'changePercent24Hr': -5.138639185629721,\n", 926 | " 'vwap24Hr': 25.557654253578594,\n", 927 | " 'explorer': 'https://etherscan.io/token/0x4e3fbd56cd56c3e72c1403e103b45db9da5b9d2b'},\n", 928 | " {'id': 'kusama',\n", 929 | " 'rank': 72,\n", 930 | " 'symbol': 'KSM',\n", 931 | " 'name': 'Kusama',\n", 932 | " 'supply': 8470098.05726206,\n", 933 | " 'maxSupply': None,\n", 934 | " 'marketCapUsd': 1253165182.3761292,\n", 935 | " 'volumeUsd24Hr': 18358816.807377826,\n", 936 | " 'priceUsd': 147.95167351122873,\n", 937 | " 'changePercent24Hr': -2.73757295077171,\n", 938 | " 'vwap24Hr': 150.16420549564762,\n", 939 | " 'explorer': 'https://kusama.subscan.io/'},\n", 940 | " {'id': 'curve-dao-token',\n", 941 | " 'rank': 73,\n", 942 | " 'symbol': 'CRV',\n", 943 | " 'name': 'Curve DAO Token',\n", 944 | " 'supply': 447228420.1829064,\n", 945 | " 'maxSupply': 3303030299.0,\n", 946 | " 'marketCapUsd': 1233600843.2753317,\n", 947 | " 'volumeUsd24Hr': 63719724.881719366,\n", 948 | " 'priceUsd': 2.758323906988775,\n", 949 | " 'changePercent24Hr': -2.8754523494951787,\n", 950 | " 'vwap24Hr': 2.7921144080046023,\n", 951 | " 'explorer': 'https://etherscan.io/token/0xD533a949740bb3306d119CC777fa900bA034cd52'},\n", 952 | " {'id': 'amp',\n", 953 | " 'rank': 74,\n", 954 | " 'symbol': 'AMP',\n", 955 | " 'name': 'Amp',\n", 956 | " 'supply': 42227702186.0,\n", 957 | " 'maxSupply': 92547638199.0,\n", 958 | " 'marketCapUsd': 1216597019.4675834,\n", 959 | " 'volumeUsd24Hr': 4147636.787945743,\n", 960 | " 'priceUsd': 0.0288104006727349,\n", 961 | " 'changePercent24Hr': -2.534788373375179,\n", 962 | " 'vwap24Hr': 0.0290037699428604,\n", 963 | " 'explorer': None},\n", 964 | " {'id': 'kadena',\n", 965 | " 'rank': 75,\n", 966 | " 'symbol': 'KDA',\n", 967 | " 'name': 'Kadena',\n", 968 | " 'supply': 171222052.90911,\n", 969 | " 'maxSupply': 1000000000.0,\n", 970 | " 'marketCapUsd': 1202142571.917528,\n", 971 | " 'volumeUsd24Hr': 15737998.561055431,\n", 972 | " 'priceUsd': 7.020956421750548,\n", 973 | " 'changePercent24Hr': -0.372187266193922,\n", 974 | " 'vwap24Hr': 6.962275234112571,\n", 975 | " 'explorer': 'https://explorer.chainweb.com/mainnet'},\n", 976 | " {'id': 'okb',\n", 977 | " 'rank': 76,\n", 978 | " 'symbol': 'OKB',\n", 979 | " 'name': 'OKB',\n", 980 | " 'supply': 60000000.0,\n", 981 | " 'maxSupply': None,\n", 982 | " 'marketCapUsd': 1168522207.6727774,\n", 983 | " 'volumeUsd24Hr': 39940584.787713,\n", 984 | " 'priceUsd': 19.475370127879625,\n", 985 | " 'changePercent24Hr': -2.796413956617042,\n", 986 | " 'vwap24Hr': 19.670131830026254,\n", 987 | " 'explorer': 'https://etherscan.io/token/0x75231f58b43240c9718dd58b4967c5114342a86c'},\n", 988 | " {'id': 'chiliz',\n", 989 | " 'rank': 77,\n", 990 | " 'symbol': 'CHZ',\n", 991 | " 'name': 'Chiliz',\n", 992 | " 'supply': 5975068035.1118145,\n", 993 | " 'maxSupply': 8888888888.0,\n", 994 | " 'marketCapUsd': 1154806546.870172,\n", 995 | " 'volumeUsd24Hr': 32524011.45734232,\n", 996 | " 'priceUsd': 0.1932708615339744,\n", 997 | " 'changePercent24Hr': -2.861455612507592,\n", 998 | " 'vwap24Hr': 0.1965903197591106,\n", 999 | " 'explorer': 'https://etherscan.io/token/0x3506424f91fd33084466f402d5d97f05f8e3b4af'},\n", 1000 | " {'id': 'loopring',\n", 1001 | " 'rank': 78,\n", 1002 | " 'symbol': 'LRC',\n", 1003 | " 'name': 'Loopring',\n", 1004 | " 'supply': 1329247432.8078947,\n", 1005 | " 'maxSupply': 1374513896.0,\n", 1006 | " 'marketCapUsd': 1142894988.1990328,\n", 1007 | " 'volumeUsd24Hr': 70325549.05337013,\n", 1008 | " 'priceUsd': 0.8598060526510011,\n", 1009 | " 'changePercent24Hr': -3.506998204211851,\n", 1010 | " 'vwap24Hr': 0.8860184922760433,\n", 1011 | " 'explorer': 'https://etherscan.io/token/0xEF68e7C694F40c8202821eDF525dE3782458639f'},\n", 1012 | " {'id': 'basic-attention-token',\n", 1013 | " 'rank': 79,\n", 1014 | " 'symbol': 'BAT',\n", 1015 | " 'name': 'Basic Attention Token',\n", 1016 | " 'supply': 1495585511.4473505,\n", 1017 | " 'maxSupply': 1500000000.0,\n", 1018 | " 'marketCapUsd': 1110049942.9874225,\n", 1019 | " 'volumeUsd24Hr': 28651977.955496904,\n", 1020 | " 'priceUsd': 0.7422176361638951,\n", 1021 | " 'changePercent24Hr': -2.3840035888232243,\n", 1022 | " 'vwap24Hr': 0.7533185929922678,\n", 1023 | " 'explorer': 'https://etherscan.io/token/Bat'},\n", 1024 | " {'id': 'celo',\n", 1025 | " 'rank': 80,\n", 1026 | " 'symbol': 'CELO',\n", 1027 | " 'name': 'Celo',\n", 1028 | " 'supply': 407486151.0,\n", 1029 | " 'maxSupply': 1000000000.0,\n", 1030 | " 'marketCapUsd': 1089607768.5306737,\n", 1031 | " 'volumeUsd24Hr': 16065421.461298868,\n", 1032 | " 'priceUsd': 2.673974970331381,\n", 1033 | " 'changePercent24Hr': -3.6954879474995956,\n", 1034 | " 'vwap24Hr': 2.7146260710136976,\n", 1035 | " 'explorer': 'https://explorer.celo.org/blocks'},\n", 1036 | " {'id': 'nexo',\n", 1037 | " 'rank': 81,\n", 1038 | " 'symbol': 'NEXO',\n", 1039 | " 'name': 'Nexo',\n", 1040 | " 'supply': 560000011.0,\n", 1041 | " 'maxSupply': 1000000000.0,\n", 1042 | " 'marketCapUsd': 1079806620.306783,\n", 1043 | " 'volumeUsd24Hr': 5020456.285253336,\n", 1044 | " 'priceUsd': 1.9282260698148148,\n", 1045 | " 'changePercent24Hr': -5.652991682413588,\n", 1046 | " 'vwap24Hr': 1.9716749909429772,\n", 1047 | " 'explorer': 'https://etherscan.io/token/0xb62132e35a6c13ee1ee0f84dc5d40bad8d815206'},\n", 1048 | " {'id': 'dash',\n", 1049 | " 'rank': 82,\n", 1050 | " 'symbol': 'DASH',\n", 1051 | " 'name': 'Dash',\n", 1052 | " 'supply': 10586488.21928824,\n", 1053 | " 'maxSupply': 18900000.0,\n", 1054 | " 'marketCapUsd': 1064293930.3745064,\n", 1055 | " 'volumeUsd24Hr': 110090735.96575385,\n", 1056 | " 'priceUsd': 100.5332371159113,\n", 1057 | " 'changePercent24Hr': -5.9274998631077205,\n", 1058 | " 'vwap24Hr': 102.98456331848097,\n", 1059 | " 'explorer': 'https://explorer.dash.org'},\n", 1060 | " {'id': 'defichain',\n", 1061 | " 'rank': 83,\n", 1062 | " 'symbol': 'DFI',\n", 1063 | " 'name': 'DeFiChain',\n", 1064 | " 'supply': 300511840.0,\n", 1065 | " 'maxSupply': 1200000000.0,\n", 1066 | " 'marketCapUsd': 1061620251.9465158,\n", 1067 | " 'volumeUsd24Hr': 2580793.974903506,\n", 1068 | " 'priceUsd': 3.5327069041489874,\n", 1069 | " 'changePercent24Hr': 1.0457513454969052,\n", 1070 | " 'vwap24Hr': 3.4885577937815233,\n", 1071 | " 'explorer': 'http://explorer.defichain.io/'},\n", 1072 | " {'id': 'waves',\n", 1073 | " 'rank': 84,\n", 1074 | " 'symbol': 'WAVES',\n", 1075 | " 'name': 'Waves',\n", 1076 | " 'supply': 107538634.0,\n", 1077 | " 'maxSupply': None,\n", 1078 | " 'marketCapUsd': 1026139538.2951733,\n", 1079 | " 'volumeUsd24Hr': 34636934.05034194,\n", 1080 | " 'priceUsd': 9.542054795815737,\n", 1081 | " 'changePercent24Hr': -4.1123168919198285,\n", 1082 | " 'vwap24Hr': 9.669877508744946,\n", 1083 | " 'explorer': 'http://wavesexplorer.com/'},\n", 1084 | " {'id': 'theta-fuel',\n", 1085 | " 'rank': 85,\n", 1086 | " 'symbol': 'TFUEL',\n", 1087 | " 'name': 'Theta Fuel',\n", 1088 | " 'supply': 5301214400.0,\n", 1089 | " 'maxSupply': None,\n", 1090 | " 'marketCapUsd': 996276240.0464923,\n", 1091 | " 'volumeUsd24Hr': 7464035.997687514,\n", 1092 | " 'priceUsd': 0.1879335874524321,\n", 1093 | " 'changePercent24Hr': -3.692615984276484,\n", 1094 | " 'vwap24Hr': 0.1925212883572914,\n", 1095 | " 'explorer': 'https://explorer.thetatoken.org/'},\n", 1096 | " {'id': 'secret',\n", 1097 | " 'rank': 86,\n", 1098 | " 'symbol': 'SCRT',\n", 1099 | " 'name': 'Secret',\n", 1100 | " 'supply': 163295557.0,\n", 1101 | " 'maxSupply': None,\n", 1102 | " 'marketCapUsd': 974274704.3108476,\n", 1103 | " 'volumeUsd24Hr': 2635817.722151135,\n", 1104 | " 'priceUsd': 5.966327083295032,\n", 1105 | " 'changePercent24Hr': -4.647801106744242,\n", 1106 | " 'vwap24Hr': 6.032634164427827,\n", 1107 | " 'explorer': 'https://www.puzzle.report/'},\n", 1108 | " {'id': 'paxos-standard',\n", 1109 | " 'rank': 87,\n", 1110 | " 'symbol': 'USDP',\n", 1111 | " 'name': 'Pax Dollar',\n", 1112 | " 'supply': 945642940.11,\n", 1113 | " 'maxSupply': None,\n", 1114 | " 'marketCapUsd': 949412480.7774596,\n", 1115 | " 'volumeUsd24Hr': 3460673.088205623,\n", 1116 | " 'priceUsd': 1.003986219859074,\n", 1117 | " 'changePercent24Hr': 0.2592121687331728,\n", 1118 | " 'vwap24Hr': None,\n", 1119 | " 'explorer': 'https://etherscan.io/token/0x8e870d67f660d95d5be530380d0ec0bd388289e1'},\n", 1120 | " {'id': 'arweave',\n", 1121 | " 'rank': 88,\n", 1122 | " 'symbol': 'AR',\n", 1123 | " 'name': 'Arweave',\n", 1124 | " 'supply': 33394701.0,\n", 1125 | " 'maxSupply': 66000000.0,\n", 1126 | " 'marketCapUsd': 914556459.3548677,\n", 1127 | " 'volumeUsd24Hr': 18493055.990352847,\n", 1128 | " 'priceUsd': 27.386274827101094,\n", 1129 | " 'changePercent24Hr': -4.7334980810022715,\n", 1130 | " 'vwap24Hr': 28.20554747295207,\n", 1131 | " 'explorer': 'https://viewblock.io/arweave'},\n", 1132 | " {'id': 'nem',\n", 1133 | " 'rank': 89,\n", 1134 | " 'symbol': 'XEM',\n", 1135 | " 'name': 'NEM',\n", 1136 | " 'supply': 8999999999.0,\n", 1137 | " 'maxSupply': 8999999999.0,\n", 1138 | " 'marketCapUsd': 913887423.792471,\n", 1139 | " 'volumeUsd24Hr': 9076115.87924631,\n", 1140 | " 'priceUsd': 0.1015430470993349,\n", 1141 | " 'changePercent24Hr': -1.6574068998858786,\n", 1142 | " 'vwap24Hr': 0.1024241567915167,\n", 1143 | " 'explorer': 'http://nembex.nem.ninja/'},\n", 1144 | " {'id': 'mina',\n", 1145 | " 'rank': 90,\n", 1146 | " 'symbol': 'MINA',\n", 1147 | " 'name': 'Mina',\n", 1148 | " 'supply': 382430492.84003925,\n", 1149 | " 'maxSupply': None,\n", 1150 | " 'marketCapUsd': 899582444.9658024,\n", 1151 | " 'volumeUsd24Hr': 15289490.276360635,\n", 1152 | " 'priceUsd': 2.3522769805442123,\n", 1153 | " 'changePercent24Hr': -5.21988205356067,\n", 1154 | " 'vwap24Hr': 2.4204175543481,\n", 1155 | " 'explorer': 'https://minaexplorer.com/'},\n", 1156 | " {'id': 'ecomi',\n", 1157 | " 'rank': 91,\n", 1158 | " 'symbol': 'OMI',\n", 1159 | " 'name': 'ECOMI',\n", 1160 | " 'supply': 166285821196.0,\n", 1161 | " 'maxSupply': 750000000000.0,\n", 1162 | " 'marketCapUsd': 892054436.7501088,\n", 1163 | " 'volumeUsd24Hr': 4866831.201637925,\n", 1164 | " 'priceUsd': 0.0053645850880975,\n", 1165 | " 'changePercent24Hr': -6.565522717966829,\n", 1166 | " 'vwap24Hr': 0.005666578550543,\n", 1167 | " 'explorer': 'https://explorer.gochain.io/addr/0x5347FDeA6AA4d7770B31734408Da6d34a8a07BdF'},\n", 1168 | " {'id': 'oasis-network',\n", 1169 | " 'rank': 92,\n", 1170 | " 'symbol': 'ROSE',\n", 1171 | " 'name': 'Oasis Network',\n", 1172 | " 'supply': 3493014306.28,\n", 1173 | " 'maxSupply': 10000000000.0,\n", 1174 | " 'marketCapUsd': 888923331.1629136,\n", 1175 | " 'volumeUsd24Hr': 51686439.31670723,\n", 1176 | " 'priceUsd': 0.254486026457075,\n", 1177 | " 'changePercent24Hr': -4.226324557124899,\n", 1178 | " 'vwap24Hr': 0.2625636130371189,\n", 1179 | " 'explorer': 'https://www.oasisscan.com/'},\n", 1180 | " {'id': 'symbol',\n", 1181 | " 'rank': 93,\n", 1182 | " 'symbol': 'XYM',\n", 1183 | " 'name': 'Symbol',\n", 1184 | " 'supply': 5582460004.558402,\n", 1185 | " 'maxSupply': 8999999999.0,\n", 1186 | " 'marketCapUsd': 881444208.9735616,\n", 1187 | " 'volumeUsd24Hr': 1390959.5492995665,\n", 1188 | " 'priceUsd': 0.1578953021165958,\n", 1189 | " 'changePercent24Hr': -2.2721238315348553,\n", 1190 | " 'vwap24Hr': 0.1600317935584702,\n", 1191 | " 'explorer': 'http://explorer.symbolblockchain.io/'},\n", 1192 | " {'id': 'holo',\n", 1193 | " 'rank': 94,\n", 1194 | " 'symbol': 'HOT',\n", 1195 | " 'name': 'Holo',\n", 1196 | " 'supply': 173208990224.84674,\n", 1197 | " 'maxSupply': None,\n", 1198 | " 'marketCapUsd': 828315144.4393038,\n", 1199 | " 'volumeUsd24Hr': 7884908.256778118,\n", 1200 | " 'priceUsd': 0.0047821717762112,\n", 1201 | " 'changePercent24Hr': -4.302111613467984,\n", 1202 | " 'vwap24Hr': 0.0048644333308558,\n", 1203 | " 'explorer': 'https://etherscan.io/token/0x6c6ee5e31d828de241282b9606c8e98ea48526e2'},\n", 1204 | " {'id': 'decred',\n", 1205 | " 'rank': 95,\n", 1206 | " 'symbol': 'DCR',\n", 1207 | " 'name': 'Decred',\n", 1208 | " 'supply': 13781571.23418176,\n", 1209 | " 'maxSupply': 21000000.0,\n", 1210 | " 'marketCapUsd': 817475116.2273432,\n", 1211 | " 'volumeUsd24Hr': 2421610.2231918084,\n", 1212 | " 'priceUsd': 59.316539626468675,\n", 1213 | " 'changePercent24Hr': 1.2615825536826268,\n", 1214 | " 'vwap24Hr': 59.77492802517528,\n", 1215 | " 'explorer': 'https://mainnet.decred.org/'},\n", 1216 | " {'id': 'iotex',\n", 1217 | " 'rank': 96,\n", 1218 | " 'symbol': 'IOTX',\n", 1219 | " 'name': 'IoTeX',\n", 1220 | " 'supply': 9540779324.30788,\n", 1221 | " 'maxSupply': 10000000000.0,\n", 1222 | " 'marketCapUsd': 802989053.4507176,\n", 1223 | " 'volumeUsd24Hr': 16445772.651241552,\n", 1224 | " 'priceUsd': 0.0841638849569523,\n", 1225 | " 'changePercent24Hr': -2.839255551511279,\n", 1226 | " 'vwap24Hr': 0.0860567923333923,\n", 1227 | " 'explorer': 'https://etherscan.io/token/0x6fb3e0a217407efff7ca062d46c26e5d60a14d69'},\n", 1228 | " {'id': 'compound',\n", 1229 | " 'rank': 97,\n", 1230 | " 'symbol': 'COMP',\n", 1231 | " 'name': 'Compound',\n", 1232 | " 'supply': 6572559.28515661,\n", 1233 | " 'maxSupply': 10000000.0,\n", 1234 | " 'marketCapUsd': 786621411.983355,\n", 1235 | " 'volumeUsd24Hr': 18629722.20610015,\n", 1236 | " 'priceUsd': 119.68266513165602,\n", 1237 | " 'changePercent24Hr': -3.158758216743868,\n", 1238 | " 'vwap24Hr': 121.6809715675423,\n", 1239 | " 'explorer': 'https://etherscan.io/token/0xc00e94cb662c3520282e6f5717214004a7f26888'},\n", 1240 | " {'id': 'yearn-finance',\n", 1241 | " 'rank': 98,\n", 1242 | " 'symbol': 'YFI',\n", 1243 | " 'name': 'yearn.finance',\n", 1244 | " 'supply': 36637.72122588,\n", 1245 | " 'maxSupply': 36666.0,\n", 1246 | " 'marketCapUsd': 782269566.8499584,\n", 1247 | " 'volumeUsd24Hr': 19790517.369040817,\n", 1248 | " 'priceUsd': 21351.479859434658,\n", 1249 | " 'changePercent24Hr': -3.6348394281036516,\n", 1250 | " 'vwap24Hr': 21772.45644955591,\n", 1251 | " 'explorer': 'https://etherscan.io/token/0x0bc529c00c6401aef6d220be8c6ea1667f6ad93e'},\n", 1252 | " {'id': 'celsius',\n", 1253 | " 'rank': 99,\n", 1254 | " 'symbol': 'CEL',\n", 1255 | " 'name': 'Celsius',\n", 1256 | " 'supply': 238863519.826,\n", 1257 | " 'maxSupply': 695658160.0,\n", 1258 | " 'marketCapUsd': 747880781.7559088,\n", 1259 | " 'volumeUsd24Hr': 1129349.7352643544,\n", 1260 | " 'priceUsd': 3.130996237101012,\n", 1261 | " 'changePercent24Hr': 0.0897325678185379,\n", 1262 | " 'vwap24Hr': 3.144216118238928,\n", 1263 | " 'explorer': 'https://etherscan.io/token/0xaaaebe6fe48e54f431b0c390cfaf0b017d09d42d'},\n", 1264 | " {'id': 'xinfin-network',\n", 1265 | " 'rank': 100,\n", 1266 | " 'symbol': 'XDC',\n", 1267 | " 'name': 'XinFin Network',\n", 1268 | " 'supply': 12293293518.55,\n", 1269 | " 'maxSupply': None,\n", 1270 | " 'marketCapUsd': 677513604.3326694,\n", 1271 | " 'volumeUsd24Hr': 4225206.656694049,\n", 1272 | " 'priceUsd': 0.0551124565040551,\n", 1273 | " 'changePercent24Hr': 2.9178435325032552,\n", 1274 | " 'vwap24Hr': 0.0558739116267885,\n", 1275 | " 'explorer': 'https://etherscan.io/token/0x41ab1b6fcbb2fa9dced81acbdec13ea6315f2bf2'}]" 1276 | ] 1277 | }, 1278 | "execution_count": 8, 1279 | "metadata": {}, 1280 | "output_type": "execute_result" 1281 | } 1282 | ], 1283 | "source": [ 1284 | "def avro_value_schema_to_python_types(inputs: dict):\n", 1285 | " schema = value_schema.to_json()[\"fields\"]\n", 1286 | "\n", 1287 | " avro_schema = {}\n", 1288 | "\n", 1289 | " for dic in schema:\n", 1290 | " avro_schema[dic[\"name\"]] = dic[\"type\"]\n", 1291 | "\n", 1292 | " avro_to_python_types = {\n", 1293 | " \"int\": int,\n", 1294 | " \"string\": str,\n", 1295 | " \"long\": float,\n", 1296 | " \"float\": float,\n", 1297 | " \"dobule\": float,\n", 1298 | " \"bytes\": bytes \n", 1299 | " }\n", 1300 | " \n", 1301 | " avro_value = None\n", 1302 | "\n", 1303 | " for key, val in inputs.items(): \n", 1304 | " avro_value = avro_schema[key] if not isinstance(avro_schema[key], list) else avro_schema[key][0] \n", 1305 | " inputs[key] = avro_to_python_types[avro_value](inputs[key]) if inputs[key] is not None else None\n", 1306 | " \n", 1307 | " return inputs\n", 1308 | " \n", 1309 | "data = [avro_value_schema_to_python_types(data_dic) for data_dic in data]\n", 1310 | "data" 1311 | ] 1312 | }, 1313 | { 1314 | "cell_type": "code", 1315 | "execution_count": 11, 1316 | "metadata": {}, 1317 | "outputs": [ 1318 | { 1319 | "name": "stdout", 1320 | "output_type": "stream", 1321 | "text": [ 1322 | "Successfully producing record value - {'id': 'xinfin-network', 'rank': 100, 'symbol': 'XDC', 'name': 'XinFin Network', 'supply': 12293293518.55, 'maxSupply': None, 'marketCapUsd': 677513604.3326694, 'volumeUsd24Hr': 4225206.656694049, 'priceUsd': 0.0551124565040551, 'changePercent24Hr': 2.9178435325032552, 'vwap24Hr': 0.0558739116267885, 'explorer': 'https://etherscan.io/token/0x41ab1b6fcbb2fa9dced81acbdec13ea6315f2bf2'} to topic - test\n" 1323 | ] 1324 | }, 1325 | { 1326 | "data": { 1327 | "text/plain": [ 1328 | "0" 1329 | ] 1330 | }, 1331 | "execution_count": 11, 1332 | "metadata": {}, 1333 | "output_type": "execute_result" 1334 | } 1335 | ], 1336 | "source": [ 1337 | "producer_config = {\n", 1338 | " \"bootstrap.servers\": bootstrap_servers,\n", 1339 | " \"schema.registry.url\": schema_registry\n", 1340 | "}\n", 1341 | "\n", 1342 | "producer = AvroProducer(producer_config, default_key_schema=key_schema, default_value_schema=value_schema)\n", 1343 | "\n", 1344 | "try:\n", 1345 | " for values in data:\n", 1346 | " record_key = str(uuid.uuid4())\n", 1347 | " producer.produce(topic=topic, key=record_key, value=values)\n", 1348 | "except Exception as e:\n", 1349 | " print(f\"Exception while producing record value - {values} to topic - {topic}: {e}\")\n", 1350 | "else:\n", 1351 | " print(f\"Successfully producing record value - {values} to topic - {topic}\")\n", 1352 | " \n", 1353 | "producer.flush()" 1354 | ] 1355 | }, 1356 | { 1357 | "cell_type": "code", 1358 | "execution_count": 2, 1359 | "metadata": {}, 1360 | "outputs": [ 1361 | { 1362 | "name": "stdout", 1363 | "output_type": "stream", 1364 | "text": [ 1365 | "Successfully producing record values - count: 100 to topic - test\n" 1366 | ] 1367 | } 1368 | ], 1369 | "source": [ 1370 | "import requests\n", 1371 | "import json\n", 1372 | "import uuid\n", 1373 | "import os\n", 1374 | "\n", 1375 | "from confluent_kafka.avro import AvroProducer\n", 1376 | "from confluent_kafka import avro\n", 1377 | "\n", 1378 | "from map_avro_types import avro_value_schema_to_python_types\n", 1379 | "\n", 1380 | "TOPIC = os.getenv(\"TOPIC\")\n", 1381 | "BOOTSTRAP_SERVERS = os.getenv(\"BOOTSTRAP_SERVERS\")\n", 1382 | "SCHEMA_REGISTRY = os.getenv(\"SCHEMA_REGISTRY\")\n", 1383 | "SCHEMA_FILE = os.getenv(\"SCHEMA_FILE\")\n", 1384 | "KEY_SCHEMA_STRING = os.getenv(\"KEY_SCHEMA_STRING\")\n", 1385 | "CRYPTO_API_URL_ASSETS = os.getenv(\"CRYPTO_API_URL_ASSETS\")\n", 1386 | "\n", 1387 | "payload = {}\n", 1388 | "headers = {}\n", 1389 | "\n", 1390 | "key_schema = avro.loads(KEY_SCHEMA_STRING)\n", 1391 | "value_schema = avro.load(SCHEMA_FILE)\n", 1392 | "\n", 1393 | "producer_config = {\n", 1394 | " \"bootstrap.servers\": BOOTSTRAP_SERVERS,\n", 1395 | " \"schema.registry.url\": SCHEMA_REGISTRY,\n", 1396 | "}\n", 1397 | "\n", 1398 | "producer = AvroProducer(\n", 1399 | " producer_config, default_key_schema=key_schema, default_value_schema=value_schema\n", 1400 | ")\n", 1401 | "\n", 1402 | "\n", 1403 | "def main():\n", 1404 | " response = requests.request(\n", 1405 | " \"GET\", CRYPTO_API_URL_ASSETS, headers=headers, data=payload\n", 1406 | " )\n", 1407 | "\n", 1408 | " status = response.status_code\n", 1409 | "\n", 1410 | " if response.status_code == 200:\n", 1411 | " data = json.loads(response.text)[\"data\"]\n", 1412 | " else:\n", 1413 | " print(f\"Request Exception {status}\")\n", 1414 | "\n", 1415 | " data = [avro_value_schema_to_python_types(data_dic, value_schema) for data_dic in data]\n", 1416 | "\n", 1417 | " try:\n", 1418 | " for values in data:\n", 1419 | " record_key = str(uuid.uuid4())\n", 1420 | " producer.produce(topic=TOPIC, key=record_key, value=values)\n", 1421 | " except Exception as e:\n", 1422 | " print(\n", 1423 | " f\"Exception while producing record value - {values} to topic - {TOPIC}: {e}\"\n", 1424 | " )\n", 1425 | " else:\n", 1426 | " print(f\"Successfully producing record values - count: {len(data)} to topic - {TOPIC}\")\n", 1427 | "\n", 1428 | " producer.flush()\n", 1429 | " \n", 1430 | "main()" 1431 | ] 1432 | }, 1433 | { 1434 | "cell_type": "code", 1435 | "execution_count": null, 1436 | "metadata": {}, 1437 | "outputs": [], 1438 | "source": [] 1439 | } 1440 | ], 1441 | "metadata": { 1442 | "kernelspec": { 1443 | "display_name": "Python 3", 1444 | "language": "python", 1445 | "name": "python3" 1446 | }, 1447 | "language_info": { 1448 | "codemirror_mode": { 1449 | "name": "ipython", 1450 | "version": 3 1451 | }, 1452 | "file_extension": ".py", 1453 | "mimetype": "text/x-python", 1454 | "name": "python", 1455 | "nbconvert_exporter": "python", 1456 | "pygments_lexer": "ipython3", 1457 | "version": "3.8.6" 1458 | } 1459 | }, 1460 | "nbformat": 4, 1461 | "nbformat_minor": 5 1462 | } 1463 | -------------------------------------------------------------------------------- /delta_spark_jupyter/src/spark-sql-kafka-0-10_2.12-3.1.2.jar: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/santiagomj/Big-Data-Streaming-Stack/d163ff9eea9c6ce7ff013cc39c6431b59f9bea3e/delta_spark_jupyter/src/spark-sql-kafka-0-10_2.12-3.1.2.jar -------------------------------------------------------------------------------- /delta_spark_jupyter/src/spark-streaming-kafka-0-10_2.13-3.2.1.jar: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/santiagomj/Big-Data-Streaming-Stack/d163ff9eea9c6ce7ff013cc39c6431b59f9bea3e/delta_spark_jupyter/src/spark-streaming-kafka-0-10_2.13-3.2.1.jar -------------------------------------------------------------------------------- /delta_spark_jupyter/src/spark_streaming_delta.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": { 7 | "tags": [] 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "import pyspark\n", 12 | "from delta import *\n", 13 | "from confluent_kafka.schema_registry import SchemaRegistryClient\n", 14 | "from pyspark.sql.avro.functions import from_avro\n", 15 | "from pyspark.sql.functions import expr\n", 16 | "\n", 17 | "import os\n", 18 | "os.environ['PYSPARK_SUBMIT_ARGS'] = '--packages org.apache.spark:spark-avro_2.12:3.0.1,org.apache.spark:spark-sql-kafka-0-10_2.12:3.0.1,org.apache.spark:spark-streaming-kafka-0-10_2.12:3.0.1,io.delta:delta-core_2.12:0.8.0,com.amazonaws:aws-java-sdk:1.11.950,org.apache.hadoop:hadoop-aws:3.2.0,net.java.dev.jets3t:jets3t:0.9.4 pyspark-shell'\n", 19 | "\n", 20 | "builder = pyspark.sql.SparkSession.builder \\\n", 21 | " .appName(\"kafka-delta-app\") \\\n", 22 | " .config(\"spark.sql.extensions\", \"io.delta.sql.DeltaSparkSessionExtension\") \\\n", 23 | " .config(\"spark.sql.catalog.spark_catalog\", \"org.apache.spark.sql.delta.catalog.DeltaCatalog\") \\\n", 24 | " .config(\"fs.s3a.path.style.access\", True) \\\n", 25 | " .config(\"fs.s3a.impl\", \"org.apache.hadoop.fs.s3a.S3AFileSystem\") \\\n", 26 | " .config(\"com.amazonaws.services.s3.enableV4\", True) \\\n", 27 | " .config(\"spark.driver.extraJavaOptions\", \"-Dcom.amazonaws.services.s3.enableV4=true\") \\\n", 28 | " .config(\"fs.s3a.access.key\", \"crypto-prices\") \\\n", 29 | " .config(\"fs.s3a.secret.key\", \"crypto-prices\") \\\n", 30 | " .config(\"fs.s3a.endpoint\", \"http://minio:9000\")\n", 31 | "\n", 32 | "spark = configure_spark_with_delta_pip(builder).getOrCreate()\n", 33 | "\n", 34 | "spark.conf.set(\"spark.sql.adaptive.enabled\", False)" 35 | ] 36 | }, 37 | { 38 | "cell_type": "code", 39 | "execution_count": 3, 40 | "metadata": { 41 | "tags": [] 42 | }, 43 | "outputs": [], 44 | "source": [ 45 | "schema_registry_conf = {'url': \"http://schema-registry:8081\"}\n", 46 | "\n", 47 | "schema_registry_client = SchemaRegistryClient(schema_registry_conf)\n", 48 | "schema_response = schema_registry_client.get_latest_version(\"crypto-prices\" + \"-value\").schema\n", 49 | "schema = schema_response.schema_str\n", 50 | "\n", 51 | "kafka_topic_name = \"crypto-prices\"\n", 52 | "kafka_bootstrap_servers = 'kafka:9092'\n", 53 | "\n", 54 | "dsraw = spark \\\n", 55 | " .readStream \\\n", 56 | " .format(\"kafka\") \\\n", 57 | " .option(\"kafka.bootstrap.servers\", kafka_bootstrap_servers) \\\n", 58 | " .option(\"subscribe\", kafka_topic_name) \\\n", 59 | " .option(\"startingOffsets\", \"earliest\") \\\n", 60 | " .option(\"failOnDataLoss\", \"false\") \\\n", 61 | " .load()\n", 62 | "\n", 63 | "from_avro_options= {\"mode\":\"PERMISSIVE\"}\n", 64 | "\n", 65 | "ds = (\n", 66 | " dsraw\n", 67 | " .select(from_avro(expr(\"substring(value, 6, length(value)-5)\"), schema, from_avro_options).alias(\"value\"))\n", 68 | " .selectExpr(\n", 69 | " \"value.id\", \n", 70 | " \"value.rank\", \n", 71 | " \"value.symbol\", \n", 72 | " \"value.name\", \n", 73 | " \"value.supply\", \n", 74 | " \"value.maxSupply\", \n", 75 | " \"value.marketCapUsd\", \n", 76 | " \"value.volumeUsd24Hr\", \n", 77 | " \"value.priceUsd\", \n", 78 | " \"value.changePercent24Hr\", \n", 79 | " \"value.vwap24Hr\", \n", 80 | " \"value.explorer\") \\\n", 81 | ")\n", 82 | "\n", 83 | "delta_table = ds.writeStream \\\n", 84 | " .format(\"delta\") \\\n", 85 | " .outputMode(\"append\") \\\n", 86 | " .option(\"checkpointLocation\", \"/delta/events/_checkpoints/etl-from-json\") \\\n", 87 | " .start(\"s3a://crypto-prices/events\")" 88 | ] 89 | }, 90 | { 91 | "cell_type": "code", 92 | "execution_count": 13, 93 | "metadata": {}, 94 | "outputs": [ 95 | { 96 | "data": { 97 | "text/plain": [ 98 | "13500" 99 | ] 100 | }, 101 | "execution_count": 13, 102 | "metadata": {}, 103 | "output_type": "execute_result" 104 | } 105 | ], 106 | "source": [ 107 | "crypto_prices = spark.read.format(\"delta\").load(\"s3a://crypto-prices/events\")\n", 108 | "crypto_prices.count()" 109 | ] 110 | }, 111 | { 112 | "cell_type": "code", 113 | "execution_count": 12, 114 | "metadata": { 115 | "tags": [] 116 | }, 117 | "outputs": [], 118 | "source": [ 119 | "# Leer y ver en consola en real time el streaming\n", 120 | "# stream2 = spark.readStream.format(\"delta\").load(\"events\").writeStream.format(\"console\").start()" 121 | ] 122 | }, 123 | { 124 | "cell_type": "code", 125 | "execution_count": null, 126 | "metadata": {}, 127 | "outputs": [], 128 | "source": [] 129 | } 130 | ], 131 | "metadata": { 132 | "kernelspec": { 133 | "display_name": "Python 3", 134 | "language": "python", 135 | "name": "python3" 136 | }, 137 | "language_info": { 138 | "codemirror_mode": { 139 | "name": "ipython", 140 | "version": 3 141 | }, 142 | "file_extension": ".py", 143 | "mimetype": "text/x-python", 144 | "name": "python", 145 | "nbconvert_exporter": "python", 146 | "pygments_lexer": "ipython3", 147 | "version": "3.8.6" 148 | } 149 | }, 150 | "nbformat": 4, 151 | "nbformat_minor": 5 152 | } 153 | -------------------------------------------------------------------------------- /docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: '3' 2 | 3 | services: 4 | zookeeper: 5 | image: zookeeper:3.7.0 6 | container_name: zookeeper 7 | ports: 8 | - "2181:2181" 9 | 10 | kafka: 11 | depends_on: 12 | - zookeeper 13 | image: wurstmeister/kafka 14 | container_name: kafka 15 | ports: 16 | - "9092:9092" 17 | - "29092:29092" 18 | environment: 19 | KAFKA_BROKER_ID: 1 20 | KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT 21 | KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092 22 | KAFKA_ADVERTISED_HOST_NAME: localhost 23 | KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 24 | KAFKA_CREATE_TOPICS: "crypto-prices:1:1" 25 | 26 | schema-registry: 27 | container_name: schema-registry 28 | image: confluentinc/cp-schema-registry:5.5.3 29 | depends_on: 30 | - zookeeper 31 | - kafka 32 | ports: 33 | - "8081:8081" 34 | environment: 35 | SCHEMA_REGISTRY_HOST_NAME: schema-registry 36 | SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: "zookeeper:2181" 37 | 38 | control-center: 39 | image: confluentinc/cp-enterprise-control-center:5.4.0 40 | hostname: control-center 41 | container_name: control-center 42 | depends_on: 43 | - zookeeper 44 | - kafka 45 | - schema-registry 46 | ports: 47 | - "9021:9021" 48 | environment: 49 | CONTROL_CENTER_BOOTSTRAP_SERVERS: 'kafka:9092' 50 | CONTROL_CENTER_ZOOKEEPER_CONNECT: 'zookeeper:2181' 51 | CONTROL_CENTER_SCHEMA_REGISTRY_URL: "http://schema-registry:8081" 52 | CONTROL_CENTER_REPLICATION_FACTOR: 1 53 | CONTROL_CENTER_INTERNAL_TOPICS_PARTITIONS: 1 54 | CONTROL_CENTER_MONITORING_INTERCEPTOR_TOPIC_PARTITIONS: 1 55 | CONFLUENT_METRICS_TOPIC_REPLICATION: 1 56 | PORT: 9021 57 | 58 | delta_pyspark: 59 | build: 60 | context: . 61 | dockerfile: delta_spark_jupyter/delta_spark.Dockerfile 62 | container_name: delta_pyspark 63 | volumes: 64 | - ./delta_spark_jupyter/src:/src 65 | ports: 66 | - "8888:8888" 67 | restart: on-failure 68 | command: /bin/sh -c "jupyter lab --ip='0.0.0.0' --port=8888 --NotebookApp.token='' --no-browser --allow-root" 69 | env_file: 70 | - producer/producer_confs.env 71 | 72 | producer_service: 73 | restart: on-failure 74 | container_name: producer_service 75 | depends_on: 76 | - createbuckets 77 | - schema-registry 78 | build: 79 | context: producer 80 | dockerfile: producer_service.Dockerfile 81 | env_file: 82 | - producer/producer_confs.env 83 | 84 | minio: 85 | container_name: minio 86 | restart: "always" 87 | image: minio/minio:RELEASE.2022-02-17T23-22-26Z 88 | env_file: 89 | - minio/minio.env 90 | ports: 91 | - "9000:9000" 92 | - "9001:9001" 93 | command: minio server --console-address ":9001" /minio_data_lake 94 | 95 | createbuckets: 96 | image: minio/mc:RELEASE.2022-02-23T03-15-59Z 97 | depends_on: 98 | - minio 99 | entrypoint: > 100 | /bin/sh -c " 101 | /usr/bin/mc alias set myminio http://minio:9000 crypto-prices crypto-prices; 102 | /usr/bin/mc mb myminio/crypto-prices; 103 | /usr/bin/mc policy set public myminio/crypto-prices; 104 | exit 0; 105 | " 106 | 107 | # Hadoop 108 | namenode: 109 | container_name: namenode 110 | image: johannestang/hadoop-namenode:2.0.0-hadoop2.8.5-java8 111 | restart: always 112 | environment: 113 | - CLUSTER_NAME=bigdata1 114 | env_file: 115 | - hive_presto/hadoop-hive.env 116 | ports: 117 | - "50070:50070" 118 | 119 | datanode: 120 | container_name: datanode 121 | image: johannestang/hadoop-datanode:2.0.0-hadoop2.8.5-java8 122 | restart: always 123 | env_file: 124 | - hive_presto/hadoop-hive.env 125 | environment: 126 | SERVICE_PRECONDITION: "namenode:50070" 127 | ports: 128 | - "50075:50075" 129 | 130 | hive-server: 131 | container_name: hive-server 132 | image: johannestang/hive:2.3.6-postgresql-metastore-s3 133 | restart: always 134 | env_file: 135 | - hive_presto/hadoop-hive.env 136 | environment: 137 | HIVE_CORE_CONF_javax_jdo_option_ConnectionURL: "jdbc:postgresql://hive-metastore/metastore" 138 | SERVICE_PRECONDITION: "hive-metastore:9083" 139 | HDFS_CONF_fs_s3a_access_key: "crypto-prices" 140 | HDFS_CONF_fs_s3a_secret_key: "crypto-prices" 141 | ports: 142 | - "10000:10000" 143 | - "10002:10002" 144 | 145 | hive-metastore: 146 | container_name: hive-metastore 147 | image: johannestang/hive:2.3.6-postgresql-metastore-s3 148 | restart: always 149 | env_file: 150 | - hive_presto/hadoop-hive.env 151 | command: /opt/hive/bin/hive --service metastore 152 | environment: 153 | SERVICE_PRECONDITION: "namenode:50070 datanode:50075 hive-metastore-postgresql:5432" 154 | HDFS_CONF_fs_s3a_access_key: "crypto-prices" 155 | HDFS_CONF_fs_s3a_secret_key: "crypto-prices" 156 | ports: 157 | - "9083:9083" 158 | hive-metastore-postgresql: 159 | container_name: hive-metastore-postgresql 160 | image: bde2020/hive-metastore-postgresql:2.3.0 161 | 162 | presto-coordinator: 163 | container_name: presto-coordinator 164 | image: johannestang/prestodb:326 165 | restart: always 166 | ports: 167 | - "8080:8080" 168 | environment: 169 | S3_ACCESS_KEY: "crypto-prices" 170 | S3_SECRET_KEY: "crypto-prices" 171 | S3_ENDPOINT: "http://minio:9000" 172 | volumes: 173 | - ./hive_presto/presto:/opt/presto/etc 174 | 175 | hue: 176 | container_name: hue 177 | image: johannestang/hue:4.5.0-presto326 178 | restart: always 179 | depends_on: 180 | - hue-postgres 181 | ports: 182 | - "8889:8888" 183 | volumes: 184 | - ./hue/config/hue.ini:/usr/share/hue/desktop/conf/z-hue.ini 185 | 186 | hue-postgres: 187 | container_name: hue-postgres 188 | image: postgres:10 189 | restart: always 190 | environment: 191 | POSTGRES_DB: hue 192 | POSTGRES_PASSWORD: hue 193 | POSTGRES_USER: hue 194 | 195 | superset: 196 | container_name: superset 197 | image: amancevice/superset:0.35.1 198 | restart: always 199 | depends_on: 200 | - superset-postgres 201 | - superset-redis 202 | ports: 203 | - "8088:8088" 204 | volumes: 205 | - ./superset/config/superset.py:/etc/superset/superset.py 206 | 207 | superset-redis: 208 | container_name: superset-redis 209 | image: redis:3.2 210 | restart: always 211 | 212 | superset-postgres: 213 | container_name: superset-postgres 214 | image: postgres:10 215 | restart: always 216 | environment: 217 | POSTGRES_DB: superset 218 | POSTGRES_PASSWORD: superset 219 | POSTGRES_USER: superset 220 | 221 | -------------------------------------------------------------------------------- /docs/add.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/santiagomj/Big-Data-Streaming-Stack/d163ff9eea9c6ce7ff013cc39c6431b59f9bea3e/docs/add.png -------------------------------------------------------------------------------- /docs/bar.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/santiagomj/Big-Data-Streaming-Stack/d163ff9eea9c6ce7ff013cc39c6431b59f9bea3e/docs/bar.png -------------------------------------------------------------------------------- /docs/config.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/santiagomj/Big-Data-Streaming-Stack/d163ff9eea9c6ce7ff013cc39c6431b59f9bea3e/docs/config.png -------------------------------------------------------------------------------- /docs/creden.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/santiagomj/Big-Data-Streaming-Stack/d163ff9eea9c6ce7ff013cc39c6431b59f9bea3e/docs/creden.png -------------------------------------------------------------------------------- /docs/data.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/santiagomj/Big-Data-Streaming-Stack/d163ff9eea9c6ce7ff013cc39c6431b59f9bea3e/docs/data.png -------------------------------------------------------------------------------- /docs/hive.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/santiagomj/Big-Data-Streaming-Stack/d163ff9eea9c6ce7ff013cc39c6431b59f9bea3e/docs/hive.png -------------------------------------------------------------------------------- /docs/pbi.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/santiagomj/Big-Data-Streaming-Stack/d163ff9eea9c6ce7ff013cc39c6431b59f9bea3e/docs/pbi.png -------------------------------------------------------------------------------- /docs/success.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/santiagomj/Big-Data-Streaming-Stack/d163ff9eea9c6ce7ff013cc39c6431b59f9bea3e/docs/success.png -------------------------------------------------------------------------------- /hive_presto/hadoop-hive.env: -------------------------------------------------------------------------------- 1 | HIVE_SITE_CONF_javax_jdo_option_ConnectionURL=jdbc:postgresql://hive-metastore-postgresql/metastore 2 | HIVE_SITE_CONF_javax_jdo_option_ConnectionDriverName=org.postgresql.Driver 3 | HIVE_SITE_CONF_javax_jdo_option_ConnectionUserName=hive 4 | HIVE_SITE_CONF_javax_jdo_option_ConnectionPassword=hive 5 | HIVE_SITE_CONF_datanucleus_autoCreateSchema=false 6 | HIVE_SITE_CONF_hive_metastore_uris=thrift://hive-metastore:9083 7 | HDFS_CONF_dfs_namenode_datanode_registration_ip___hostname___check=false 8 | 9 | CORE_CONF_fs_defaultFS=hdfs://namenode:8020 10 | CORE_CONF_hadoop_http_staticuser_user=root 11 | CORE_CONF_hadoop_proxyuser_hue_hosts=* 12 | CORE_CONF_hadoop_proxyuser_hue_groups=* 13 | 14 | HDFS_CONF_dfs_webhdfs_enabled=true 15 | HDFS_CONF_dfs_permissions_enabled=false 16 | 17 | YARN_CONF_yarn_log___aggregation___enable=true 18 | YARN_CONF_yarn_resourcemanager_recovery_enabled=true 19 | YARN_CONF_yarn_resourcemanager_store_class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore 20 | YARN_CONF_yarn_resourcemanager_fs_state___store_uri=/rmstate 21 | YARN_CONF_yarn_nodemanager_remote___app___log___dir=/app-logs 22 | YARN_CONF_yarn_log_server_url=http://historyserver:8188/applicationhistory/logs/ 23 | YARN_CONF_yarn_timeline___service_enabled=true 24 | YARN_CONF_yarn_timeline___service_generic___application___history_enabled=true 25 | YARN_CONF_yarn_resourcemanager_system___metrics___publisher_enabled=true 26 | YARN_CONF_yarn_resourcemanager_hostname=resourcemanager 27 | YARN_CONF_yarn_timeline___service_hostname=historyserver 28 | YARN_CONF_yarn_resourcemanager_address=resourcemanager:8032 29 | YARN_CONF_yarn_resourcemanager_scheduler_address=resourcemanager:8030 30 | YARN_CONF_yarn_resourcemanager_resource__tracker_address=resourcemanager:8031 31 | 32 | HDFS_CONF_fs_s3a_endpoint=http://minio:9000 33 | HDFS_CONF_fs_s3a_path_style_access=true 34 | HDFS_CONF_fs_s3a_impl=org.apache.hadoop.fs.s3a.S3AFileSystem 35 | -------------------------------------------------------------------------------- /hive_presto/presto/catalog/hive.properties: -------------------------------------------------------------------------------- 1 | connector.name=hive-hadoop2 2 | hive.metastore.uri=thrift://hive-metastore:9083 3 | hive.metastore-timeout=1m 4 | hive.s3.aws-access-key=crypto-prices 5 | hive.s3.aws-secret-key=crypto-prices 6 | hive.s3.endpoint=http://minio:9000 7 | hive.s3.path-style-access=true 8 | hive.s3.ssl.enabled=false 9 | -------------------------------------------------------------------------------- /hive_presto/presto/config.properties: -------------------------------------------------------------------------------- 1 | coordinator=true 2 | node-scheduler.include-coordinator=true 3 | http-server.http.port=8080 4 | query.max-memory=5GB 5 | query.max-memory-per-node=1GB 6 | discovery-server.enabled=true 7 | discovery.uri=http://presto-coordinator:8080 8 | -------------------------------------------------------------------------------- /hive_presto/presto/jvm.config: -------------------------------------------------------------------------------- 1 | -server 2 | -Xmx16G 3 | -XX:+UseG1GC 4 | -XX:G1HeapRegionSize=32M 5 | -XX:+UseGCOverheadLimit 6 | -XX:+ExplicitGCInvokesConcurrent 7 | -XX:+HeapDumpOnOutOfMemoryError 8 | -XX:OnOutOfMemoryError=kill -9 %p 9 | -------------------------------------------------------------------------------- /hive_presto/presto/node.properties: -------------------------------------------------------------------------------- 1 | node.environment=development 2 | node.id=presto-coordinator 3 | node.data-dir=/opt/presto/data 4 | -------------------------------------------------------------------------------- /hue/config/hue.ini: -------------------------------------------------------------------------------- 1 | # Lightweight Hue configuration file 2 | # ================================== 3 | 4 | [desktop] 5 | 6 | # Set this to a random string, the longer the better. 7 | secret_key=kasdlfjknasdfl3hbaksk3bwkasdfkasdfba23asdf 8 | 9 | # Webserver listens on this address and port 10 | http_host=0.0.0.0 11 | http_port=8888 12 | 13 | # Time zone name 14 | time_zone=Europe/Copenhagen 15 | 16 | # Enable or disable debug mode. 17 | django_debug_mode=false 18 | 19 | # Enable or disable backtrace for server error 20 | http_500_debug_mode=false 21 | 22 | app_blacklist=search,hbase,impala,jobbrowser,jobsub,pig,security,spark,sqoop,zookeeper 23 | 24 | # This should be the hadoop cluster admin 25 | default_hdfs_superuser=root 26 | 27 | # Configuration options for specifying the Desktop Database. For more info, 28 | # see http://docs.djangoproject.com/en/1.11/ref/settings/#database-engine 29 | # ------------------------------------------------------------------------ 30 | [[database]] 31 | # Database engine is typically one of: 32 | # postgresql_psycopg2, mysql, sqlite3 or oracle. 33 | # 34 | # Note that for sqlite3, 'name', below is a path to the filename. For other backends, it is the database name 35 | # Note for Oracle, options={"threaded":true} must be set in order to avoid crashes. 36 | # Note for Oracle, you can use the Oracle Service Name by setting "host=" and "port=" and then "name=:/". 37 | # Note for MariaDB use the 'mysql' engine. 38 | 39 | engine=postgresql_psycopg2 40 | host=hue-postgres 41 | port=5432 42 | user=hue 43 | password=hue 44 | name=hue 45 | 46 | # engine=mysql 47 | # host=database 48 | # port=3306 49 | # user=root 50 | # password=secret 51 | # name=hue 52 | 53 | ########################################################################### 54 | # Settings to configure the snippets available in the Notebook 55 | ########################################################################### 56 | 57 | [notebook] 58 | 59 | # One entry for each type of snippet. 60 | [[interpreters]] 61 | # Define the name and how to connect and execute the language. 62 | # https://docs.gethue.com/administrator/configuration/editor/ 63 | 64 | # Example for Docker compose 65 | # [[[mysql]]] 66 | # name = MySQL 67 | # interface=sqlalchemy 68 | # ## https://docs.sqlalchemy.org/en/latest/dialects/mysql.html 69 | # options='{"url": "mysql://root:secret@database:3306/hue"}' 70 | # ## options='{"url": "mysql://${USER}:${PASSWORD}@localhost:3306/hue"}' 71 | 72 | [[[hive]]] 73 | name=Hive 74 | interface=hiveserver2 75 | 76 | # [[[impala]]] 77 | # name=Impala 78 | # interface=hiveserver2 79 | 80 | # [[[sql]]] 81 | # name=SparkSql 82 | # interface=livy 83 | 84 | # [[[spark]]] 85 | # name=Scala 86 | # interface=livy 87 | 88 | # [[[pyspark]]] 89 | # name=PySpark 90 | # interface=livy 91 | 92 | # [[[r]]] 93 | # name=R 94 | # interface=livy 95 | 96 | # [[jar]]] 97 | # name=Spark Submit Jar 98 | # interface=livy-batch 99 | 100 | # [[[py]]] 101 | # name=Spark Submit Python 102 | # interface=livy-batch 103 | 104 | # [[[text]]] 105 | # name=Text 106 | # interface=text 107 | 108 | # [[[markdown]]] 109 | # name=Markdown 110 | # interface=text 111 | 112 | # [[[sqlite]]] 113 | # name = SQLite 114 | # interface=rdbms 115 | 116 | # [[[postgresql]]] 117 | # name = PostgreSQL 118 | # interface=rdbms 119 | 120 | # [[[oracle]]] 121 | # name = Oracle 122 | # interface=rdbms 123 | 124 | # [[[solr]]] 125 | # name = Solr SQL 126 | # interface=solr 127 | # ## Name of the collection handler 128 | # # options='{"collection": "default"}' 129 | 130 | # [[[pig]]] 131 | # name=Pig 132 | # interface=oozie 133 | 134 | # [[[java]]] 135 | # name=Java 136 | # interface=oozie 137 | 138 | # [[[spark2]]] 139 | # name=Spark 140 | # interface=oozie 141 | 142 | # [[[mapreduce]]] 143 | # name=MapReduce 144 | # interface=oozie 145 | 146 | # [[[sqoop1]]] 147 | # name=Sqoop1 148 | # interface=oozie 149 | 150 | # [[[distcp]]] 151 | # name=Distcp 152 | # interface=oozie 153 | 154 | # [[[shell]]] 155 | # name=Shell 156 | # interface=oozie 157 | 158 | [[[presto]]] 159 | name=Presto SQL 160 | interface=presto 161 | ## Specific options for connecting to the Presto server. 162 | ## The JDBC driver presto-jdbc.jar need to be in the CLASSPATH environment variable. 163 | ## If 'user' and 'password' are omitted, they will be prompted in the UI. 164 | options='{"url": "jdbc:presto://presto-coordinator:8080/hive/default", "driver": "io.prestosql.jdbc.PrestoDriver", "user": "", "password": ""}' 165 | 166 | # [[[clickhouse]]] 167 | # name=ClickHouse 168 | # interface=jdbc 169 | # ## Specific options for connecting to the ClickHouse server. 170 | # ## The JDBC driver clickhouse-jdbc.jar and its related jars need to be in the CLASSPATH environment variable. 171 | # options='{"url": "jdbc:clickhouse://localhost:8123", "driver": "ru.yandex.clickhouse.ClickHouseDriver", "user": "readonly", "password": ""}' 172 | 173 | 174 | [dashboard] 175 | 176 | # Activate the SQL Dashboard (beta). 177 | has_sql_enabled=true 178 | 179 | 180 | [hadoop] 181 | 182 | # Configuration for HDFS NameNode 183 | # ------------------------------------------------------------------------ 184 | [[hdfs_clusters]] 185 | # HA support by using HttpFs 186 | 187 | [[[default]]] 188 | # Enter the filesystem uri 189 | fs_defaultfs=hdfs://namenode:8020 190 | 191 | # Use WebHdfs/HttpFs as the communication mechanism. 192 | # Domain should be the NameNode or HttpFs host. 193 | # Default port is 14000 for HttpFs. 194 | webhdfs_url=http://namenode:50070/webhdfs/v1 195 | 196 | # Configuration for YARN (MR2) 197 | # ------------------------------------------------------------------------ 198 | # [[yarn_clusters]] 199 | 200 | # [[[default]]] 201 | # Enter the host on which you are running the ResourceManager 202 | ## resourcemanager_host=localhost 203 | 204 | # The port where the ResourceManager IPC listens on 205 | ## resourcemanager_port=8032 206 | 207 | # URL of the ResourceManager API 208 | ## resourcemanager_api_url=http://localhost:8088 209 | 210 | # URL of the ProxyServer API 211 | ## proxy_api_url=http://localhost:8088 212 | 213 | # URL of the HistoryServer API 214 | ## history_server_api_url=http://localhost:19888 215 | 216 | # URL of the Spark History Server 217 | ## spark_history_server_url=http://localhost:18088 218 | 219 | 220 | ########################################################################### 221 | # Settings to configure Beeswax with Hive 222 | ########################################################################### 223 | 224 | [beeswax] 225 | 226 | # Host where HiveServer2 is running. 227 | # If Kerberos security is enabled, use fully-qualified domain name (FQDN). 228 | hive_server_host=hive-server 229 | 230 | # Port where HiveServer2 Thrift server runs on. 231 | ## hive_server_port=10000 232 | 233 | 234 | ########################################################################### 235 | # Settings to configure Impala 236 | ########################################################################### 237 | 238 | [impala] 239 | # Host of the Impala Server (one of the Impalad) 240 | ## server_host=localhost 241 | 242 | # Port of the Impala Server 243 | ## server_port=21050 244 | 245 | 246 | ########################################################################### 247 | # Settings to configure the Spark application. 248 | ########################################################################### 249 | 250 | [spark] 251 | # The Livy Server URL. 252 | ## livy_server_url=http://localhost:8998 253 | 254 | # Configure Livy to start in local 'process' mode, or 'yarn' workers. 255 | ## livy_server_session_kind=yarn 256 | 257 | # Whether Livy requires client to perform Kerberos authentication. 258 | ## security_enabled=false 259 | 260 | # Host of the Sql Server 261 | ## sql_server_host=localhost 262 | 263 | # Port of the Sql Server 264 | ## sql_server_port=10000 265 | 266 | # Choose whether Hue should validate certificates received from the server. 267 | ## ssl_cert_ca_verify=true 268 | 269 | 270 | ########################################################################### 271 | # Settings to configure HBase Browser 272 | ########################################################################### 273 | 274 | [hbase] 275 | # Comma-separated list of HBase Thrift servers for clusters in the format of '(name|host:port)'. 276 | ## hbase_clusters=(Cluster|localhost:9090) 277 | 278 | 279 | ########################################################################### 280 | # Settings to configure Solr Search 281 | ########################################################################### 282 | 283 | [search] 284 | 285 | # URL of the Solr Server 286 | ## solr_url=http://localhost:8983/solr/ 287 | 288 | 289 | ########################################################################### 290 | # Settings to configure liboozie 291 | ########################################################################### 292 | 293 | [liboozie] 294 | # The URL where the Oozie service runs on. This is required in order for 295 | # users to submit jobs. Empty value disables the config check. 296 | ## oozie_url=http://localhost:11000/oozie 297 | 298 | 299 | ########################################################################### 300 | # Settings for the AWS lib 301 | ########################################################################### 302 | 303 | [aws] 304 | [[aws_accounts]] 305 | # Default AWS account 306 | ## [[[default]]] 307 | # AWS credentials 308 | ## access_key_id= 309 | ## secret_access_key= 310 | ## security_token= 311 | 312 | # Execute this script to produce the AWS access key ID. 313 | ## access_key_id_script=/path/access_key_id.sh 314 | 315 | # Execute this script to produce the AWS secret access key. 316 | ## secret_access_key_script=/path/secret_access_key.sh 317 | 318 | # Allow to use either environment variables or 319 | # EC2 InstanceProfile to retrieve AWS credentials. 320 | ## allow_environment_credentials=yes 321 | 322 | # AWS region to use, if no region is specified, will attempt to connect to standard s3.amazonaws.com endpoint 323 | ## region=us-east-1 324 | 325 | # Endpoint overrides 326 | ## host= 327 | 328 | # Proxy address and port 329 | ## proxy_address= 330 | ## proxy_port=8080 331 | ## proxy_user= 332 | ## proxy_pass= 333 | 334 | # Secure connections are the default, but this can be explicitly overridden: 335 | ## is_secure=true 336 | 337 | 338 | ########################################################################### 339 | # Settings for the Azure lib 340 | ########################################################################### 341 | [azure] 342 | [[azure_accounts]] 343 | # Default Azure account 344 | [[[default]]] 345 | # Azure credentials 346 | ## client_id= 347 | # Execute this script to produce the ADLS client id. 348 | ## client_id_script=/path/client_id.sh 349 | ## client_secret= 350 | # Execute this script to produce the ADLS client secret. 351 | ## client_secret_script=/path/client_secret.sh 352 | ## tenant_id= 353 | # Execute this script to produce the ADLS tenant id. 354 | ## tenant_id_script=/path/tenant_id.sh 355 | 356 | [[adls_clusters]] 357 | # Default ADLS cluster 358 | [[[default]]] 359 | ## fs_defaultfs=adl://.azuredatalakestore.net 360 | ## webhdfs_url=https://.azuredatalakestore.net/webhdfs/v1 361 | 362 | 363 | ########################################################################### 364 | # Settings to configure the ZooKeeper Lib 365 | ########################################################################### 366 | 367 | [libzookeeper] 368 | # ZooKeeper ensemble. Comma separated list of Host/Port. 369 | # e.g. localhost:2181,localhost:2182,localhost:2183 370 | ## ensemble=localhost:2181 371 | 372 | 373 | ########################################################################### 374 | # Settings to configure Kafka 375 | ########################################################################### 376 | 377 | [kafka] 378 | 379 | [[kafka]] 380 | # Enable the Kafka integration. 381 | ## is_enabled=false 382 | 383 | # Base URL of Kafka REST API. 384 | ## api_url=http://localhost:8082 385 | 386 | 387 | ########################################################################### 388 | # Settings to configure Metadata 389 | ########################################################################### 390 | 391 | [metadata] 392 | 393 | [[navigator]] 394 | # Navigator API URL (without version suffix). 395 | ## api_url=http://localhost:7187/api 396 | 397 | # Which authentication to use: CM or external via LDAP or SAML. 398 | ## navmetadataserver_auth_type=CMDB 399 | 400 | # Username of the CM user used for authentication. 401 | ## navmetadataserver_cmdb_user=hue 402 | # CM password of the user used for authentication. 403 | ## navmetadataserver_cmdb_password= 404 | # Execute this script to produce the CM password. This will be used when the plain password is not set. 405 | # navmetadataserver_cmdb_password_script= 406 | -------------------------------------------------------------------------------- /minio/minio.env: -------------------------------------------------------------------------------- 1 | MINIO_ROOT_USER=crypto-prices 2 | MINIO_ROOT_PASSWORD=crypto-prices 3 | MINIO_HOST=minio 4 | MINIO_PORT=9000 5 | -------------------------------------------------------------------------------- /producer/.dockerignore: -------------------------------------------------------------------------------- 1 | producer_confs.env -------------------------------------------------------------------------------- /producer/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/santiagomj/Big-Data-Streaming-Stack/d163ff9eea9c6ce7ff013cc39c6431b59f9bea3e/producer/__init__.py -------------------------------------------------------------------------------- /producer/__pycache__/map_avro_types.cpython-38.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/santiagomj/Big-Data-Streaming-Stack/d163ff9eea9c6ce7ff013cc39c6431b59f9bea3e/producer/__pycache__/map_avro_types.cpython-38.pyc -------------------------------------------------------------------------------- /producer/avro/crypto-schema.avsc: -------------------------------------------------------------------------------- 1 | { 2 | "namespace": "io.codebrews.cryptoschema", 3 | "type": "record", 4 | "name": "CryptoSchema", 5 | "fields": [ 6 | { 7 | "name": "id", 8 | "type": "string" 9 | }, 10 | { 11 | "name": "rank", 12 | "type": "int" 13 | }, 14 | { 15 | "name": "symbol", 16 | "type": "string" 17 | }, 18 | { 19 | "name": "name", 20 | "type": "string" 21 | }, 22 | { 23 | "name": "supply", 24 | "type": ["float", "null"] 25 | }, 26 | { 27 | "name": "maxSupply", 28 | "type": ["float", "null"] 29 | }, 30 | { 31 | "name": "marketCapUsd", 32 | "type": ["float", "null"] 33 | }, 34 | { 35 | "name": "volumeUsd24Hr", 36 | "type": ["float", "null"] 37 | }, 38 | { 39 | "name": "priceUsd", 40 | "type": ["float", "null"] 41 | }, 42 | { 43 | "name": "changePercent24Hr", 44 | "type": ["float", "null"] 45 | }, 46 | { 47 | "name": "vwap24Hr", 48 | "type": ["float", "null"] 49 | }, 50 | { 51 | "name": "explorer", 52 | "type": ["string", "null"] 53 | } 54 | ] 55 | } -------------------------------------------------------------------------------- /producer/crypto_producer.py: -------------------------------------------------------------------------------- 1 | import requests 2 | import json 3 | import uuid 4 | import os 5 | from time import sleep 6 | 7 | from confluent_kafka.avro import AvroProducer 8 | from confluent_kafka import avro 9 | 10 | from map_avro_types import avro_value_schema_to_python_types 11 | 12 | TOPIC = os.getenv("TOPIC") 13 | BOOTSTRAP_SERVERS = os.getenv("BOOTSTRAP_SERVERS") 14 | SCHEMA_REGISTRY = os.getenv("SCHEMA_REGISTRY") 15 | SCHEMA_FILE = os.getenv("SCHEMA_FILE") 16 | KEY_SCHEMA_STRING = os.getenv("KEY_SCHEMA_STRING") 17 | CRYPTO_API_URL_ASSETS = os.getenv("CRYPTO_API_URL_ASSETS") 18 | TIME_TO_SLEEP = int(os.getenv("REQUEST_DELAY")) 19 | 20 | payload = {} 21 | headers = {} 22 | 23 | key_schema = avro.loads(KEY_SCHEMA_STRING) 24 | value_schema = avro.load(SCHEMA_FILE) 25 | 26 | producer_config = { 27 | "bootstrap.servers": BOOTSTRAP_SERVERS, 28 | "schema.registry.url": SCHEMA_REGISTRY, 29 | } 30 | 31 | producer = AvroProducer( 32 | producer_config, default_key_schema=key_schema, default_value_schema=value_schema 33 | ) 34 | 35 | 36 | def main(): 37 | response = requests.request( 38 | "GET", CRYPTO_API_URL_ASSETS, headers=headers, data=payload 39 | ) 40 | 41 | status = response.status_code 42 | 43 | if response.status_code == 200: 44 | data = json.loads(response.text)["data"] 45 | else: 46 | print(f"Request Exception {status}") 47 | return 48 | 49 | data = [ 50 | avro_value_schema_to_python_types(data_dic, value_schema) for data_dic in data 51 | ] 52 | 53 | try: 54 | for values in data: 55 | record_key = str(uuid.uuid4()) 56 | producer.produce(topic=TOPIC, key=record_key, value=values) 57 | except Exception as e: 58 | print( 59 | f"Exception while producing record value - {values} to topic - {TOPIC}: {e}" 60 | ) 61 | else: 62 | print( 63 | f"Successfully producing record values - count: {len(data)} to topic - {TOPIC}" 64 | ) 65 | 66 | producer.flush() 67 | 68 | 69 | if __name__ == "__main__": 70 | while True: 71 | main() 72 | sleep(TIME_TO_SLEEP) -------------------------------------------------------------------------------- /producer/map_avro_types.py: -------------------------------------------------------------------------------- 1 | def avro_value_schema_to_python_types(inputs: dict, value_schema): 2 | schema = value_schema.to_json()["fields"] 3 | 4 | avro_schema = {} 5 | 6 | for dic in schema: 7 | avro_schema[dic["name"]] = dic["type"] 8 | 9 | avro_to_python_types = { 10 | "int": int, 11 | "string": str, 12 | "long": float, 13 | "float": float, 14 | "dobule": float, 15 | "bytes": bytes 16 | } 17 | 18 | avro_value = None 19 | 20 | for key, val in inputs.items(): 21 | avro_value = avro_schema[key] if not isinstance(avro_schema[key], list) else avro_schema[key][0] 22 | inputs[key] = avro_to_python_types[avro_value](inputs[key]) if inputs[key] is not None else None 23 | 24 | return inputs -------------------------------------------------------------------------------- /producer/producer_confs.env: -------------------------------------------------------------------------------- 1 | TOPIC=crypto-prices 2 | BOOTSTRAP_SERVERS=kafka:9092 3 | SCHEMA_REGISTRY=http://schema-registry:8081 4 | SCHEMA_FILE=./avro/crypto-schema.avsc 5 | KEY_SCHEMA_STRING={"type": "string"} 6 | CRYPTO_API_URL_ASSETS=https://api.coincap.io/v2/assets 7 | REQUEST_DELAY=30 8 | -------------------------------------------------------------------------------- /producer/producer_service.Dockerfile: -------------------------------------------------------------------------------- 1 | FROM python:3.9.10 2 | 3 | WORKDIR /usr/app/src 4 | COPY . ./ 5 | 6 | ARG DEBIAN_FRONTEND=noninteractive 7 | 8 | RUN cat /etc/os-release 9 | 10 | RUN apt-get update && apt-get install software-properties-common -y 11 | 12 | RUN wget -qO - http://packages.confluent.io/deb/3.2/archive.key | apt-key add - 13 | 14 | RUN add-apt-repository "deb [arch=amd64] http://packages.confluent.io/deb/3.2 stable main" 15 | RUN apt-get update && apt-get install confluent-platform-oss-2.11 -y 16 | 17 | RUN apt-get update && apt install librdkafka-dev -y 18 | 19 | RUN apt-get update && pip install -r producer_service_requirements.txt 20 | 21 | CMD [ "python", "-u", "./crypto_producer.py"] -------------------------------------------------------------------------------- /producer/producer_service_requirements.txt: -------------------------------------------------------------------------------- 1 | confluent-kafka[avro]==1.3.0 -------------------------------------------------------------------------------- /superset/config/superset_config.py: -------------------------------------------------------------------------------- 1 | import os 2 | 3 | MAPBOX_API_KEY = os.getenv('MAPBOX_API_KEY', '') 4 | CACHE_CONFIG = { 5 | 'CACHE_TYPE': 'redis', 6 | 'CACHE_DEFAULT_TIMEOUT': 300, 7 | 'CACHE_KEY_PREFIX': 'superset_', 8 | 'CACHE_REDIS_HOST': 'superset-redis', 9 | 'CACHE_REDIS_PORT': 6379, 10 | 'CACHE_REDIS_DB': 1, 11 | 'CACHE_REDIS_URL': 'redis://superset-redis:6379/1'} 12 | SQLALCHEMY_DATABASE_URI = \ 13 | 'postgresql+psycopg2://superset:superset@superset-postgres:5432/superset' 14 | SQLALCHEMY_TRACK_MODIFICATIONS = True 15 | SECRET_KEY = 'thisISaSECRET_1234' 16 | --------------------------------------------------------------------------------