├── department_etl.png ├── README.md └── noharm-nifi-template.xml /department_etl.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/noharm-ai/nifi-docker/HEAD/department_etl.png -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Nifi-Docker 2 | NiFi docker container customization from the [official Apache Nifi Docker Image](https://hub.docker.com/r/apache/nifi) 3 | 4 | ### 1. Build Docker 5 | Pull the latest version of Apache Nifi Docker, build and run 6 | 7 | ```shell 8 | $ docker pull apache/nifi:1.27.0 9 | ``` 10 | Option 1: Run without authentication 11 | ```shell 12 | $ docker run --log-opt max-size=100m --name noharm-nifi -e NIFI_WEB_HTTP_PORT='8080' -p 8080:8080 -d apache/nifi:1.27.0 --restart=always 13 | ``` 14 | Open the URL http://localhost:8080/nifi. 15 | 16 | Option 2: Run with authentication 17 | ```shell 18 | docker run --log-opt max-size=100m --name noharm-nifi -e NIFI_WEB_HTTPS_PORT='8443' -p 8443:8443 -d -e SINGLE_USER_CREDENTIALS_USERNAME=nifi_noharm -e SINGLE_USER_CREDENTIALS_PASSWORD=nifi_pass apache/nifi:1.27.0 --restart=always 19 | ``` 20 | Watch until the container is ready. You are looking for the "NiFi has started. The UI is available" message. 21 | ```shell 22 | docker logs -f noharm-nifi 23 | ``` 24 | Open the URL https://localhost:8443/nifi 25 | 26 | ### 2. Download Connectors and Add Timezone 27 | 28 | Dive into container shell, add timezone and download JARs & NARs 29 | ```shell 30 | $ docker exec --user="root" -it nifi /bin/bash 31 | $ echo "java.arg.8=-Duser.timezone=America/Sao_Paulo" >> conf/bootstrap.conf 32 | $ wget https://raw.githubusercontent.com/noharm-ai/nifi-composer/refs/heads/main/genkeypair.sh 33 | $ chmod +x genkeypair.sh 34 | $ ./genkeypair.sh 35 | $ cd lib 36 | ``` 37 | Copy & Paste it 38 | ```shell 39 | wget https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.30/mysql-connector-java-8.0.30.jar 40 | wget https://repo1.maven.org/maven2/com/oracle/database/jdbc/ojdbc8/23.2.0.0/ojdbc8-23.2.0.0.jar 41 | wget https://jdbc.postgresql.org/download/postgresql-42.7.3.jar 42 | wget https://repo1.maven.org/maven2/org/apache/nifi/nifi-kite-nar/1.15.3/nifi-kite-nar-1.15.3.nar 43 | wget https://truststore.pki.rds.amazonaws.com/sa-east-1/sa-east-1-bundle.pem 44 | ``` 45 | - Updated JDBC PostgreSQL Driver at https://jdbc.postgresql.org/download 46 | - Updated JDBC Oracle Driver at https://www.oracle.com/database/technologies/appdev/jdbc-downloads.html 47 | - Updated JDBC MySQL Driver at https://dev.mysql.com/downloads/connector/j/ 48 | 49 | Configure aws 50 | ```shell 51 | $ docker exec --user="root" -it noharm-nifi apt update 52 | $ docker exec --user="root" -it noharm-nifi apt install nano vim awscli -y 53 | $ exit 54 | $ docker exec --user="nifi" -it noharm-nifi /bin/bash 55 | $ aws configure 56 | Access key ID and Secret access key: available at https://noharm.odoo.com/odoo/knowledge/34/knowledge/72/knowledge/92 57 | Region: sa-east-1 58 | Output format: None 59 | $ aws s3 ls #para testar o acesso 60 | $ exit 61 | $ docker restart noharm-nifi 62 | ``` 63 | 64 | ### 4. Access Nifi Web Service 65 | 66 | Now you can access your Nifi web service at http://localhost:8080/nifi/ or https://localhost:8443/nifi/ 67 | 68 | If you are in a VPN environment remember to tunnel 8080 port 69 | ```shell 70 | $ ssh -f user@ip_address -L 8080:localhost:8080 -N 71 | ``` 72 | [How to tunnel in Putty](https://blog.devolutions.net/2017/4/how-to-configure-an-ssh-tunnel-on-putty) (for Windows users) 73 | 74 | ### 5. Basic Nifi Usage 75 | 76 | This wiki page will cover the main proccess of NoHarm integration: 77 | [Basic Nifi Usage](https://github.com/noharm-ai/nifi-docker/wiki/Basic-Nifi-Usage) 78 | 79 | ### Troubleshooting 80 | 81 | #### a. Testing JDBC Connection 82 | 83 | - Download tool from http://jdbcsql.sourceforge.net/ 84 | - Run: java -jar jdbcsql-1.0.zip -h hostname -p 1521 -U user -P pass -d service -m oracle "SELECT table_name FROM all_tables" 85 | 86 | #### b. Install SQL Plus 87 | 88 | - Install: 89 | ```shell 90 | sudo yum install -y yum install https://download.oracle.com/otn_software/linux/instantclient/216000/oracle-instantclient-basic-21.6.0.0.0-1.el8.x86_64.rpm 91 | sudo yum install -y yum install https://download.oracle.com/otn_software/linux/instantclient/216000/oracle-instantclient-sqlplus-21.6.0.0.0-1.el8.x86_64.rpm 92 | ``` 93 | - Run: 94 | ```shell 95 | sqlplus user/pass@localhost:1521/service 96 | ``` 97 | 98 | SQLPlus for Debian: https://askubuntu.com/questions/159939/how-to-install-sqlplus 99 | 100 | #### c. Repair "no space left" issue 101 | 102 | Prune useless container and images: 103 | ```shell 104 | $ docker system prune -a 105 | ``` 106 | 107 | Check fuller folders (if possible): 108 | ```shell 109 | $ docker exec -t nifi sh -c "du -sh *" 110 | ``` 111 | Empty target folder (usually content_repository): 112 | ```shell 113 | $docker exec -t nifi sh -c "rm content_repository/* -rf" 114 | ``` 115 | Restart nifi to repair empty folder: 116 | ```shell 117 | docker exec -t nifi sh -c "./bin/nifi.sh restart" 118 | ``` 119 | 120 | You can do the same inside the container: 121 | ```shell 122 | $ docker exec --user="root" -it nifi /bin/bash 123 | ``` 124 | #### d. Fix "docker.sock permission denied" issue 125 | ```shell 126 | sudo chown $USER /var/run/docker.sock 127 | ``` 128 | 129 | #### e. Restarting Docker service if Docker is down: 130 | Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? 131 | ```shell 132 | sudo service docker start 133 | ``` 134 | 135 | #### e. Changing Properties from HTTPS to HTTP: 136 | 137 | - https://github.com/apache/nifi/blob/main/nifi-docker/dockerhub/sh/start.sh#L53-L69 138 | -------------------------------------------------------------------------------- /noharm-nifi-template.xml: -------------------------------------------------------------------------------- 1 | 2 |