├── .gitignore ├── Main instructable.md ├── README.md ├── ansible ├── files │ ├── usr │ │ └── lib │ │ │ └── cgi-bin │ │ │ ├── favicon.ico │ │ │ ├── monitor.py │ │ │ ├── webgui.py │ │ │ └── webgui.py.save │ └── var │ │ └── www │ │ ├── favicon.png │ │ └── templog.db ├── inventory.py ├── main.yml ├── playbooks │ └── upgrade.yml ├── pyrun.py ├── templates │ └── vhosts2.conf.j2 └── vars │ ├── apache_packages.yml │ └── path.yml ├── module description.txt └── rpi ├── rpi_detector.py ├── rpi_detector.pyc └── rpi_ssh.ssh /.gitignore: -------------------------------------------------------------------------------- 1 | 2 | ansible/files/ssh/authorized_keys 3 | -------------------------------------------------------------------------------- /Main instructable.md: -------------------------------------------------------------------------------- 1 | 0 Raspberry Pi orchestration 2 | 3 | In this Instructable I would like to share my personal work-flow for working with headless Raspberry Pi. 4 | If you are interested in automating the setup and deployment process for one Pi or many this is the post for you. 5 | All the code is available on github: ​Raspberry Pi orchestration 6 | Please give feedback as this is my first post and first python project. 7 | Many thanks to ​HackBergen club and ​Verkstedet hackerspace in Bergen, Norway for help, space and hardware to test my project. 8 | Thanks to ​Instructables for sending Raspery Pi 2 in quantity which allowed to create big network for testing. 9 | 10 | 1 Short history 11 | 12 | ​cover picture link to preserve copyright on it [http://www.element14.com/community/servlet/JiveServlet/showImage/38-14865-192066/Raspberry%2BPi%2B%2BB%2B%2B(2).jpg] 13 | 14 | I love RPi as a fast way to solve personal problems. Make a media center, maybe a internet radio, NAS or whatever. 15 | 16 | I never tried running any desktop on RPi or connected keyboard to it. But to use it this way is not always easy. The fastest way I found was to use Google Coder project. It was so fast. Burn the image to SD card and connect ethernet cable or WiFi dongle and you are good to go. But as all raspbian solutions it needs 4 Gb sd card. I try to re use old cards or make projects as small as possible. 17 | 18 | What I did for myself is a Frankenstein of different projects to make it possible to find all the raspberry pi's on the network, install any distro of your choice to sd card or USB flashdrive and them provision it as you need. And if you have apps to deploy you can do it too. 19 | 20 | 2 What you need 21 | 22 | Raspberry pi, any model will do, this script can be changed to run on all of them. But it is much simpler if you don't use A or A+ because they don't have Ethernet port. 23 | Internet. and fast one if you don't want to wait long/ For first runs all the files will be downloaded from external sources. 24 | Any computer running OS X, Linux or Windows. All steps can be reproduced on any of the platforms, some modifications to the code needed. 25 | Know basic python. 26 | This tutorial is not for the people who only starting with Raspberry Pi. 27 | 28 | Try 29 | 30 | - https://www.raspberrypi.org/ 31 | 32 | - https://googlecreativelab.github.io/coder/ 33 | 34 | - http://www.instructables.com/tag/type-id/category-... 35 | 36 | for guides for first start. 37 | 38 | 3 Sturcture 39 | 40 | This os a big project for me, so I'll have to dived it into parts. 41 | 42 | here is the first draft of what I wanted to achieve: 43 | 44 | rpi_detector 45 | dicovery module, use any way to detect ip by partial mac address 46 | use the ip to create ansible inventory 47 | optional 48 | test for standard login methods # integrated in rpi_ssh 49 | ssh with password 50 | ssh with key 51 | rpi_inventory 52 | use detector to get _list_ of Rpis 53 | rpi_ssh - generate ansible inventory based on a template 54 | use templates for role definition 55 | rpi_ssh 56 | test for default ssh login methods 57 | ssh with password 58 | if true continue to switch to key, else ssh with key 59 | ssh with key : if policy mandates switch to key, else exit 60 | 61 | switch to key 62 | use pre-defined root key 63 | generate root key # if used for key deployment use this strategy - generate keys based on the _list_ for root 64 | generate keys for defined users 65 | generate authorised_keys files 66 | copy to defined folders 67 | rpi_create # will be used to deploy OS on raspberry pi with netinstaller 68 | use template to generate 69 | installer-config.txt 70 | post-install.txt 71 | copy needed files (authorized-keys, hosts, rules etc.) 72 | update/change boot-loader 73 | if local repository is used verify accessibility, update and verify needed packages 74 | restart all affected machines to start creation process 75 | - use base image if available 76 | rpi_provission 77 | if used as stand alone app: 78 | rpi_detector 79 | rpi_inventory 80 | else 81 | run update, upgrade playbook 82 | run role provision playbooks 83 | rpi_deploy 84 | playbook for environment verification 85 | playbook for deployment 86 | playbook for deployment verification 87 | rpi_monitor 88 | monitor demon for controlling cluster status 89 | failure detection and re-install 90 | Next steps will describe how I managed and sometimes failed in thees steps. 91 | 92 | I use OS X as a development platform, but tried to test all modules on Windows and Linux. I down't have full solutions for those platforms right now, hope to get them in the future. 93 | 94 | 4 OS 95 | 96 | To do anything with Raspberry Pi you need to install minimal software. 97 | 98 | OS for Raspberry Pi come in different flavors, you can find some on official website or search for "raspberry pi os". 99 | 100 | I use alternative way to get very small and custom os installed raspbian net installer. 101 | 102 | Pros: 103 | 104 | very minimal install, in minimal server configuration uses ~380 Mb. 105 | customizable. You can define packages needed. 106 | Simple install, copy files to FAT partition on SD card and plug it to RPi 107 | Can install to USB flashdrive or other media if needed 108 | It is possible, after first run, re-install OS without taking out SD card or touching RPi via ssh. 109 | Cons: 110 | 111 | Needs internet to get files from repository, if you deploy many machines it will consume lots of traffic. For this case it is recommended to have local clone of repository with necessary packages and use Ansible to add additional ones on per-machine basis. 112 | Takes time. If used over Internet speed of install will depend on downlink. 113 | You can run this once and create preferable base installation, create an image of it and use that next time. 114 | 115 | Yes, it is possible to use base image to deploy os over network.(Will add if there is interest) 116 | 117 | To define installer parameters and install additional packages use installer-config.txt. Put it on /boot partition of SD card you plug to your RPi. 118 | 119 | If you want to use Ansible next add python to your custom installer-config.txt. It would be a good thing to change root password, just in case. 120 | 121 | 5 Detector 122 | 123 | By now you should have a raspberry pi running, with os of your choice and connected to your network. 124 | 125 | There are options how to find Raspberry Pi on the network. 126 | 127 | nmap - great multi platform tool for network discovery and security auditing. You can scan your network for all connected devices and get ip addresses for RPi. Downside is that you need to install it on OS X and Windows and not all Linux distros have it pre-installed. I use it all the time for different applications and if you like it – use this option for the discovery part. 128 | login to your router if possible and look up the ip that was assigned to RPi. Not very scriptable process. 129 | if RPi has avahi-daemon running than use Apple zeroconfig, Bonjour in Windows, or avahi-browse in Linux to get ip addresses. 130 | Pi Finder.app - Simple Apple script app for discovery. 131 | many more 132 | I needed small and fast solution that needs no additional apps, libs for python – "arp". Arp is networking tool available on almost all platforms, definitely on OS X, Windows and Linux. It produces basically same output and readily available through python. 133 | 134 | You can see I found 4 RPi's on my network. 135 | 136 | There is a caveat with this code. By default arp table is empty and you have to fill it. My monitor server fills it for it constantly in contact with all machines. 137 | 138 | Possible solutions: 139 | 140 | - use nmap instead, or use it as a solution to fill the arp table . 141 | 142 | - add ping loop for the subnet scan 143 | 144 | I use arp because nmap needs additional library in python to function properly and parsing nmap output as is not as simple as arp for me. 145 | 146 | From her you can pipe it to next stage. 147 | 148 | 6 Inventory 149 | 150 | So we got ip addresses. 151 | 152 | We can use them to connect to RPi's using ssh and configure them as we want. 153 | 154 | For automating this process there are special tools: Puppet, Chef, SaltStack etc. 155 | 156 | I use Ansible because: 157 | 158 | Server only. Doesn't need client software installed 159 | Python based 160 | Very simple YAML configuration files 161 | Main concept is to ru scripts with tasks you want to perform(Playbooks) against list of machines you have (Inventory). Read more in documentation. 162 | 163 | This step explains how to make inventory file from list of IPs we got before. 164 | 165 | 7 Ansible Playbooks 166 | 167 | This part is just an example of what is possible with ansible playbooks. 168 | 169 | I will use my testing project to deploy raspberry pi temperature monitor with web interface. It uses : 170 | 171 | python script to: 172 | 173 | get temperature from ds18b20 1-wire sensors, store it in sqlite database; 174 | 175 | view a web ui with temperature plot 176 | 177 | apache server to serve web page and run cgi script to generate temperature plot. 178 | 179 | File structure of my project 180 | 181 | - project folder/ 182 | 183 | - inventory 184 | 185 | - main.yml # is the main file that I run to do all tasks, it includes update/upgrade features, provisioning and deploymnent tasks. As it is simple project there no roles and decision making. 186 | 187 | - files/ # is a folder to store all files you need to copy to raspberry pi, for example public ssh keys 188 | 189 | - playbooks/ # is a folder for ansible playbooks, if you want to have separate set of tasks for different scenarios like update/upgrade as you don't want to run it every time. 190 | 191 | - vars/ # stores files with different variables you might need like paths and permissions, user lists etc. 192 | 193 | - templates/ # stores jinja2 template files to generate different things on the machines. I use it to create apache configuration, but it's not necessary for simple setup 194 | 195 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Raspberry-Pi-orchestration 2 | basic workflow for setup, provision and deployment with python and ansible 3 | 4 | This is a small project to deploy temperature logger to raspberry pi 5 | 6 | Code for Instructable http://www.instructables.com/id/Raspberry-Pi-orchestration/ 7 | -------------------------------------------------------------------------------- /ansible/files/usr/lib/cgi-bin/favicon.ico: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TranceCat/Raspberry-Pi-orchestration/f6f6e8be0cab6ce55b906574c91fafaf5f0f30ec/ansible/files/usr/lib/cgi-bin/favicon.ico -------------------------------------------------------------------------------- /ansible/files/usr/lib/cgi-bin/monitor.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | import sqlite3 4 | import os 5 | import time 6 | import glob 7 | 8 | # global variables 9 | dbname='/var/www/templog.db' 10 | 11 | # store the temperature in the database 12 | def log_temperature(temp,w1id): 13 | 14 | conn=sqlite3.connect(dbname) 15 | curs=conn.cursor() 16 | 17 | curs.execute("INSERT INTO temps values(datetime('now','localtime'), (?),(?))", (temp,w1id)) 18 | 19 | # commit the changes 20 | conn.commit() 21 | 22 | conn.close() 23 | 24 | # display the contents of the database 25 | def display_data(): 26 | 27 | conn=sqlite3.connect(dbname) 28 | curs=conn.cursor() 29 | 30 | for row in curs.execute("SELECT * FROM temps"): 31 | print str(row[0])+" "+str(row[1]) 32 | 33 | conn.close() 34 | 35 | # get temerature 36 | # returns None on error, or the temperature as a float 37 | def get_temp(devicefile): 38 | 39 | try: 40 | fileobj = open(devicefile,'r') 41 | lines = fileobj.readlines() 42 | fileobj.close() 43 | except: 44 | return None 45 | 46 | # get the status from the end of line 1 47 | status = lines[0][-4:-1] 48 | 49 | # is the status is ok, get the temperature from line 2 50 | if status=="YES": 51 | #print status 52 | tempstr= lines[1][-6:-1] 53 | tempvalue=float(tempstr)/1000 54 | #print tempvalue 55 | return tempvalue 56 | else: 57 | #print "There was an error." 58 | return None 59 | 60 | 61 | 62 | # main function 63 | # This is where the program starts 64 | def main(): 65 | 66 | # enable kernel modules 67 | os.system('sudo modprobe w1-gpio') 68 | os.system('sudo modprobe w1-therm') 69 | 70 | # search for a device file that starts with 28 71 | devicelist = glob.glob('/sys/bus/w1/devices/28*') 72 | 73 | if devicelist=='': 74 | #print ("No device to probe.") 75 | return None 76 | else: 77 | # append /w1slave to the device file 78 | for i in devicelist: 79 | w1devicefile = i + '/w1_slave' 80 | #print ("this is "+w1devicefile[-24:-9]) 81 | temperature = get_temp(w1devicefile) 82 | if temperature != None: 83 | log_temperature(temperature,w1devicefile[-24:-9]) 84 | else: 85 | temperature = get_temp(w1devicefile) 86 | log_temperature(temperature,w1devicefile[-24:-9]) 87 | 88 | if __name__=="__main__": 89 | main() 90 | 91 | 92 | 93 | 94 | -------------------------------------------------------------------------------- /ansible/files/usr/lib/cgi-bin/webgui.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | import sqlite3 4 | import sys 5 | import cgi 6 | import cgitb 7 | 8 | 9 | # global variables 10 | speriod=(15*60)-1 11 | dbname='/var/www/templog.db' 12 | device_id_static = '28-00000448d1a2' #long wire 13 | #device_id_static = '28-000004488f83' 14 | 15 | # print the HTTP header 16 | def printHTTPheader(): 17 | print "Content-type: text/html\n\n" 18 | 19 | 20 | 21 | # print the HTML head section 22 | # arguments are the page title and the table for the chart 23 | def printHTMLHead(title, table): 24 | print "" 25 | print " " 26 | print title 27 | print " " 28 | 29 | print_graph_script(table) 30 | 31 | print "" 32 | 33 | 34 | # get data from the database 35 | # if an interval is passed, 36 | # return a list of records from the database 37 | def get_data(interval,device_id): 38 | 39 | conn=sqlite3.connect(dbname) 40 | curs=conn.cursor() 41 | 42 | if interval == None: 43 | curs.execute("SELECT * FROM temps") 44 | else: 45 | 46 | curs.execute("SELECT * FROM temps WHERE id = '%s' AND timestamp>datetime('2015-06-06','-%s hours')" % (device_id,interval)) 47 | 48 | rows=curs.fetchall() 49 | 50 | conn.close() 51 | 52 | return rows 53 | 54 | 55 | # convert rows from database into a javascript table 56 | def create_table(rows): 57 | chart_table="" 58 | 59 | for row in rows[:]: 60 | rowstr="['{0}', {1}],\n".format(str(row[0]),str(row[1])) 61 | chart_table+=rowstr 62 | #print (chart_table) 63 | #row=rows[-1] 64 | #rowstr="['{0}', {1}]\n".format(str(row[0]),str(row[1])) 65 | #chart_table+=rowstr 66 | 67 | 68 | 69 | return chart_table 70 | 71 | 72 | # print the javascript to generate the chart 73 | # pass the table generated from the database info 74 | def print_graph_script(table): 75 | 76 | # google chart snippet 77 | chart_code=""" 78 | 86 | 87 | """ 106 | 107 | print chart_code % (table) 108 | 109 | 110 | 111 | 112 | # print the div that contains the graph 113 | def show_graph(): 114 | print "

Temperature Chart

" 115 | print '
' 116 | 117 | 118 | 119 | # connect to the db and show some stats 120 | # argument option is the number of hours 121 | def show_stats(option): 122 | 123 | conn=sqlite3.connect(dbname) 124 | curs=conn.cursor() 125 | 126 | if option is None: 127 | option = str(24) 128 | 129 | curs.execute("SELECT timestamp,max(temp) FROM temps WHERE timestamp>datetime('now','-%s hour') AND timestamp<=datetime('now')" % option) 130 | rowmax=curs.fetchone() 131 | rowstrmax="{0}   {1}C".format(str(rowmax[0]),str(rowmax[1])) 132 | 133 | curs.execute("SELECT timestamp,min(temp) FROM temps WHERE timestamp>datetime('now','-%s hour') AND timestamp<=datetime('now')" % option) 134 | rowmin=curs.fetchone() 135 | rowstrmin="{0}   {1}C".format(str(rowmin[0]),str(rowmin[1])) 136 | 137 | curs.execute("SELECT avg(temp) FROM temps WHERE timestamp>datetime('now','-%s hour') AND timestamp<=datetime('now')" % option) 138 | rowavg=curs.fetchone() 139 | 140 | 141 | print "
" 142 | 143 | 144 | print "

Minumum temperature 

" 145 | print rowstrmin 146 | print "

Maximum temperature

" 147 | print rowstrmax 148 | print "

Average temperature

" 149 | print "%.3f" % rowavg+"C" 150 | 151 | print "
" 152 | 153 | print "

In the last hour:

" 154 | print "" 155 | print "" 156 | 157 | rows=curs.execute("SELECT * FROM temps WHERE timestamp>datetime('new','-1 hour') AND timestamp<=datetime('new')") 158 | for row in rows: 159 | rowstr="".format(str(row[0]),str(row[1])) 160 | print rowstr 161 | print "
Date/TimeTemperature
{0}  {1}C
" 162 | 163 | print "
" 164 | 165 | conn.close() 166 | 167 | 168 | 169 | 170 | def print_time_selector(option): 171 | 172 | print """
173 | Show the temperature logs for 174 | 200 | 201 |
""" 202 | 203 | 204 | # check that the option is valid 205 | # and not an SQL injection 206 | def validate_input(option_str): 207 | # check that the option string represents a number 208 | if option_str.isalnum(): 209 | # check that the option is within a specific range 210 | if int(option_str) > 0 and int(option_str) <= 24: 211 | return option_str 212 | else: 213 | return None 214 | else: 215 | return None 216 | 217 | 218 | #return the option passed to the script 219 | def get_option(): 220 | form=cgi.FieldStorage() 221 | if "timeinterval" in form: 222 | option = form["timeinterval"].value 223 | return validate_input (option) 224 | else: 225 | return None 226 | 227 | 228 | # main function 229 | # This is where the program starts 230 | def main(): 231 | 232 | cgitb.enable() 233 | 234 | # get options that may have been passed to this script 235 | option=get_option() 236 | 237 | if option is None: 238 | option = str(6) 239 | 240 | # get data from the database 241 | records=get_data(option,device_id_static) 242 | 243 | # print the HTTP header 244 | printHTTPheader() 245 | 246 | if len(records) != 0: 247 | # convert the data into a table 248 | table=create_table(records) 249 | else: 250 | print "No data found" 251 | return 252 | 253 | # start printing the page 254 | print "" 255 | # print the head section including the table 256 | # used by the javascript for the chart 257 | printHTMLHead("Raspberry Pi Temperature Logger", table) 258 | 259 | # print the page body 260 | print "" 261 | print "

Raspberry Pi Temperature Logger

" 262 | 263 | 264 | print "
" 265 | print_time_selector(option) 266 | show_graph() 267 | #show_stats(option) 268 | print "" 269 | print "" 270 | 271 | sys.stdout.flush() 272 | 273 | if __name__=="__main__": 274 | main() 275 | 276 | 277 | 278 | 279 | -------------------------------------------------------------------------------- /ansible/files/usr/lib/cgi-bin/webgui.py.save: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | import sqlite3 4 | import sys 5 | import cgi 6 | import cgitb 7 | 8 | 9 | # global variables 10 | speriod=(15*60)-1 11 | dbname='/var/www/templog.db' 12 | 13 | 14 | # print the HTTP header 15 | def printHTTPheader(): 16 | print "Content-type: text/html\n\n" 17 | 18 | 19 | 20 | # print the HTML head section 21 | # arguments are the page title and the table for the chart 22 | def printHTMLHead(title, table): 23 | print " " 24 | print " " 25 | print title 26 | print " " 27 | 28 | print_graph_script(table) 29 | 30 | print "" 31 | 32 | 33 | # get data from the database 34 | # if an interval is passed, 35 | # return a list of records from the database 36 | def get_data(interval): 37 | 38 | conn=sqlite3.connect(dbname) 39 | curs=conn.cursor() 40 | 41 | if interval == None: 42 | curs.execute("SELECT * FROM temps") 43 | else: 44 | curs.execute("SELECT * FROM temps WHERE timestamp>datetime('now','-%s hours')" % interval) 45 | rows=curs.fetchall() 46 | 47 | conn.close() 48 | 49 | return rows 50 | 51 | 52 | # convert rows from database into a javascript table 53 | def create_table(rows): 54 | chart_table="" 55 | 56 | for row in rows[:-1]: 57 | rowstr="['{0}', {1}],\n".format(str(row[0]),str(row[1])) 58 | chart_table+=rowstr 59 | 60 | row=rows[-1] 61 | rowstr="['{0}', {1}]\n".format(str(row[0]),str(row[1])) 62 | chart_table+=rowstr 63 | 64 | return chart_table 65 | 66 | 67 | # print the javascript to generate the chart 68 | # pass the table generated from the database info 69 | def print_graph_script(table): 70 | 71 | # google chart snippet 72 | chart_code=""" 73 | 74 | """ 91 | 92 | print chart_code % (table) 93 | 94 | 95 | 96 | 97 | # print the div that contains the graph 98 | def show_graph(): 99 | print "

Temperature Chart

" 100 | print '
' 101 | 102 | 103 | 104 | # connect to the db and show some stats 105 | # argument option is the number of hours 106 | def show_stats(option): 107 | 108 | conn=sqlite3.connect(dbname) 109 | curs=conn.cursor() 110 | 111 | if option is None: 112 | option = str(24) 113 | 114 | curs.execute("SELECT timestamp,max(temp) FROM temps WHERE timestamp>datetime('now','-%s hour') AND timestamp<=datetime('now')" % option) 115 | rowmax=curs.fetchone() 116 | rowstrmax="{0}   {1}C".format(str(rowmax[0]),str(rowmax[1])) 117 | 118 | curs.execute("SELECT timestamp,min(temp) FROM temps WHERE timestamp>datetime('now','-%s hour') AND timestamp<=datetime('now')" % option) 119 | rowmin=curs.fetchone() 120 | rowstrmin="{0}   {1}C".format(str(rowmin[0]),str(rowmin[1])) 121 | 122 | curs.execute("SELECT avg(temp) FROM temps WHERE timestamp>datetime('now','-%s hour') AND timestamp<=datetime('now')" % option) 123 | rowavg=curs.fetchone() 124 | 125 | 126 | print "
" 127 | 128 | 129 | print "

Minumum temperature 

" 130 | print rowstrmin 131 | print "

Maximum temperature

" 132 | print rowstrmax 133 | print "

Average temperature

" 134 | print "%.3f" % rowavg+"C" 135 | 136 | print "
" 137 | 138 | print "

In the last hour:

" 139 | print "" 140 | print "" 141 | 142 | rows=curs.execute("SELECT * FROM temps WHERE timestamp>datetime('new','-1 hour') AND timestamp<=datetime('new')") 143 | for row in rows: 144 | rowstr="".format(str(row[0]),str(row[1])) 145 | print rowstr 146 | print "
Date/TimeTemperature
{0}  {1}C
" 147 | 148 | print "
" 149 | 150 | conn.close() 151 | 152 | 153 | 154 | 155 | def print_time_selector(option): 156 | 157 | print """
158 | Show the temperature logs for 159 | 185 | 186 |
""" 187 | 188 | 189 | # check that the option is valid 190 | # and not an SQL injection 191 | def validate_input(option_str): 192 | # check that the option string represents a number 193 | if option_str.isalnum(): 194 | # check that the option is within a specific range 195 | if int(option_str) > 0 and int(option_str) <= 24: 196 | return option_str 197 | else: 198 | return None 199 | else: 200 | return None 201 | 202 | 203 | #return the option passed to the script 204 | def get_option(): 205 | form=cgi.FieldStorage() 206 | if "timeinterval" in form: 207 | option = form["timeinterval"].value 208 | return validate_input (option) 209 | else: 210 | return None 211 | 212 | 213 | 214 | 215 | # main function 216 | # This is where the program starts 217 | def main(): 218 | 219 | cgitb.enable() 220 | 221 | # get options that may have been passed to this script 222 | option=get_option() 223 | 224 | if option is None: 225 | option = str(24) 226 | 227 | # get data from the database 228 | records=get_data(option) 229 | 230 | # print the HTTP header 231 | printHTTPheader() 232 | 233 | if len(records) != 0: 234 | # convert the data into a table 235 | table=create_table(records) 236 | else: 237 | print "No data found" 238 | return 239 | 240 | # start printing the page 241 | print "" 242 | # print the head section including the table 243 | # used by the javascript for the chart 244 | printHTMLHead("Raspberry Pi Temperature Logger", table) 245 | 246 | # print the page body 247 | print "" 248 | print "

Raspberry Pi Temperature Logger

" 249 | print "
" 250 | print_time_selector(option) 251 | show_graph() 252 | show_stats(option) 253 | print "" 254 | print "" 255 | 256 | sys.stdout.flush() 257 | 258 | if __name__=="__main__": 259 | main() 260 | 261 | 262 | 263 | 264 | -------------------------------------------------------------------------------- /ansible/files/var/www/favicon.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TranceCat/Raspberry-Pi-orchestration/f6f6e8be0cab6ce55b906574c91fafaf5f0f30ec/ansible/files/var/www/favicon.png -------------------------------------------------------------------------------- /ansible/files/var/www/templog.db: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TranceCat/Raspberry-Pi-orchestration/f6f6e8be0cab6ce55b906574c91fafaf5f0f30ec/ansible/files/var/www/templog.db -------------------------------------------------------------------------------- /ansible/inventory.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | ''' 3 | Ansible dynamic inventory experimentation 4 | 5 | Output dynamic inventory as JSON from statically defined data structures 6 | ''' 7 | 8 | ''' 9 | ANSIBLE_INV = { 10 | "rpi": { 11 | "hosts": ["rpi1", "rpi2", "rpi3", "rpi4"], 12 | "vars": { 13 | "ansible_ssh_user": "root", 14 | "ansible_ssh_private_key_file":"~/.ssh/rpi" 15 | } 16 | } 17 | } 18 | ''' 19 | 20 | import argparse 21 | import json 22 | import sys 23 | sys.path.append('../rpi/') 24 | import rpi_detector 25 | 26 | def output_list_inventory(json_output): 27 | ''' 28 | Output the --list data structure as JSON 29 | ''' 30 | print json.dumps(json_output) 31 | 32 | 33 | def find_host(search_host, inventory): 34 | ''' 35 | Find the given variables for the given host and output them as JSON 36 | ''' 37 | host_attribs = inventory.get(search_host, {}) 38 | print json.dumps(host_attribs) 39 | 40 | 41 | def main(): 42 | # Argument parsing 43 | parser = argparse.ArgumentParser(description="Ansible dynamic inventory") 44 | parser.add_argument("--list", help="Ansible inventory of all of the groups", 45 | action="store_true", dest="list_inventory") 46 | parser.add_argument("--host", help="Ansible inventory of a particular host", action="store", 47 | dest="ansible_host", type=str) 48 | 49 | cli_args = parser.parse_args() 50 | list_inventory = cli_args.list_inventory 51 | ansible_host = cli_args.ansible_host 52 | 53 | # Finding RPi 54 | rpi_detector.run() 55 | ANSIBLE_INV= rpi_detector.var_gen_inv() 56 | HOST_VARS = rpi_detector.var_gen_host() 57 | 58 | 59 | if list_inventory: 60 | output_list_inventory(ANSIBLE_INV) 61 | 62 | if ansible_host: 63 | find_host(ansible_host, HOST_VARS) 64 | 65 | 66 | if __name__ == "__main__": 67 | main() -------------------------------------------------------------------------------- /ansible/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | #- include: playbooks/upgrade.yml 3 | - hosts: rpi 4 | gather_facts: no 5 | sudo: no 6 | 7 | vars_files: 8 | - vars/path.yml 9 | - vars/apache_packages.yml 10 | 11 | tasks: 12 | 13 | - name: Update apt cache. 14 | apt: update_cache=yes cache_valid_time=86400 15 | 16 | - name: Ensure Apache is installed. 17 | apt: "name={{ item }} state=installed" 18 | with_items: apache_packages 19 | 20 | - shell: ls -1 /etc/apache2/sites-enabled 21 | register: contents 22 | 23 | - file: path=/etc/apache2/sites-enabled/{{ item }} state=absent 24 | with_items: contents.stdout_lines 25 | 26 | - name: Copy files 27 | copy: src={{ item.src }} dest={{ item.dst }} 28 | with_items: 29 | - {src: files/usr/lib/cgi-bin/, dst: /usr/lib/cgi-bin/ } 30 | - {src: files/var/www/ , dst: /var/www/} 31 | 32 | - name: Permissions 33 | file: path={{ item.pth }} owner={{ item.own }} group={{ item.grp }} mode=0777 recurse=yes 34 | with_items: 35 | - {pth: /usr/lib/cgi-bin/ , own: www-data , grp: www-data } 36 | - {pth: /var/www/ , own: www-data , grp: www-data } 37 | - cron: name="run monitor" minute="5" user="root" job="/usr/lib/cgi-bin/monitor.py" state=absent 38 | 39 | - name: Add apache vhosts configuration. 40 | template: 41 | src: "templates/vhosts2.conf.j2" 42 | dest: "/etc/apache2/sites-available/test.conf" 43 | owner: root 44 | group: root 45 | mode: 0644 46 | 47 | - name: Add vhost symlink in sites-enabled. 48 | file: 49 | src: "/etc/apache2/sites-available/test.conf" 50 | dest: "/etc/apache2/sites-enabled/test.conf" 51 | state: link 52 | #uncomment for debian jessie 53 | # - name: activate cgi 54 | # command: a2enmod cgi 55 | 56 | - name: restart apache 57 | action: service name=apache2 state=restarted 58 | 59 | -------------------------------------------------------------------------------- /ansible/playbooks/upgrade.yml: -------------------------------------------------------------------------------- 1 | - hosts: all 2 | gather_facts: no 3 | sudo: no 4 | tasks: 5 | - name: updates a server 6 | apt: update_cache=yes cache_valid_time=3600 7 | - name: upgrade a server 8 | apt: upgrade=yes -------------------------------------------------------------------------------- /ansible/pyrun.py: -------------------------------------------------------------------------------- 1 | import ansible.runner 2 | import ansible.playbook 3 | import ansible.inventory 4 | from ansible import callbacks 5 | from ansible import utils 6 | import json 7 | 8 | 9 | utils.VERBOSITY = 0 10 | playbook_cb = callbacks.PlaybookCallbacks(verbose=utils.VERBOSITY) 11 | stats = callbacks.AggregateStats() 12 | runner_cb = callbacks.PlaybookRunnerCallbacks(stats, verbose=utils.VERBOSITY) 13 | 14 | 15 | inventory = ansible.inventory.Inventory("./inventory.py") 16 | 17 | # run modules 18 | pm = ansible.runner.Runner( 19 | module_name = 'ping', #'command', 20 | #module_args = 'uname -a', 21 | inventory = inventory, 22 | pattern = "all" 23 | ) 24 | 25 | out = pm.run() 26 | print json.dumps(out, sort_keys=True, indent=4, separators=(',', ': ')) 27 | 28 | 29 | # run playbooks 30 | #pb = ansible.playbook.PlayBook( 31 | # playbook= "./main.yml", 32 | # callbacks=playbook_cb, 33 | # runner_callbacks=runner_cb, 34 | # stats=stats, 35 | # inventory = inventory 36 | # ) 37 | #out = pb.run() 38 | # 39 | #playbook_cb.on_stats(pb.stats) 40 | #print out 41 | 42 | 43 | 44 | -------------------------------------------------------------------------------- /ansible/templates/vhosts2.conf.j2: -------------------------------------------------------------------------------- 1 | 2 | ServerAdmin webmaster@localhost 3 | 4 | DocumentRoot /var/www 5 | 6 | Options Indexes FollowSymLinks Includes ExecCGI 7 | AllowOverride None 8 | # AllowOverride All 9 | # Require all granted 10 | # Allow from all 11 | 12 | 13 | Options Indexes FollowSymLinks MultiViews 14 | AllowOverride None 15 | Order allow,deny 16 | allow from all 17 | 18 | 19 | ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/ 20 | 21 | AddHandler cgi-script .py 22 | AllowOverride None 23 | Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch 24 | Order allow,deny 25 | Allow from all 26 | 27 | 28 | ErrorLog ${APACHE_LOG_DIR}/error.log 29 | 30 | # Possible values include: debug, info, notice, warn, error, crit, 31 | # alert, emerg. 32 | LogLevel warn 33 | 34 | CustomLog ${APACHE_LOG_DIR}/access.log combined 35 | -------------------------------------------------------------------------------- /ansible/vars/apache_packages.yml: -------------------------------------------------------------------------------- 1 | --- 2 | apache_daemon: apache2 3 | apache_server_root: /etc/apache2 4 | apache_conf_path: /etc/apache2 5 | 6 | apache_packages: 7 | - apache2 8 | - apache2-mpm-prefork 9 | - apache2-utils 10 | - sqlite3 11 | - python-pip 12 | - git 13 | 14 | apache_ports_configuration_items: 15 | - { 16 | regexp: "^Listen ", 17 | line: "Listen {{ apache_listen_port }}" 18 | } -------------------------------------------------------------------------------- /ansible/vars/path.yml: -------------------------------------------------------------------------------- 1 | --- 2 | add_paths: 3 | - /usr/lib/cgi-bin/ 4 | - /var/www/ 5 | 6 | -------------------------------------------------------------------------------- /module description.txt: -------------------------------------------------------------------------------- 1 | rpi_detector 2 | - dicovery module, use any way to detect ip by partial mac address 3 | - use the ip to create ansible inventory 4 | optional 5 | - test for standard login methods # integrated in rpi_ssh 6 | - ssh with password 7 | - ssh with key 8 | ----------------------------------------------------------------------------------------------------------- 9 | 10 | rpi_inventory 11 | - use detector to get _list_ of Rpis 12 | - rpi_ssh 13 | - generate ansible inventory based on a template 14 | - use templates for role definition 15 | - 16 | ----------------------------------------------------------------------------------------------------------- 17 | rpi_ssh 18 | - test for default ssh login methods 19 | - ssh with password 20 | if true continue to switch to key, else ssh with key 21 | - ssh with key 22 | if policy mandates switch to key, else exit 23 | - switch to key 24 | - use pre-defined root key 25 | - generate root key 26 | # if used for key deployment use this strategy 27 | - generate keys based on the _list_ for root 28 | - generate keys for defined users 29 | - generate authorised_keys files 30 | - copy to defined folders 31 | ----------------------------------------------------------------------------------------------------------- 32 | rpi_create # will be used to deploy OS on raspberry pi with netinstaller 33 | - use template to generate 34 | - installer-config.txt 35 | - post-install.txt 36 | - copy needed files (authorized-keys, hosts, rules etc.) 37 | - update/change boot-loader 38 | - if local repository is used verify accessibility, update and verify needed packages 39 | - restart all affected machines to start creation process 40 | - use base image if available 41 | ----------------------------------------------------------------------------------------------------------- 42 | 43 | rpi_provission 44 | if used as stand alone app: 45 | - rpi_detector 46 | - rpi_inventory 47 | else 48 | - run update, upgrade playbook 49 | - run role provision playbooks 50 | 51 | rpi_deploy 52 | - playbook for environment verification 53 | - playbook for deployment 54 | - playbook for deployment verification 55 | 56 | rpi_monitor 57 | - monitor demon for controlling cluster status 58 | - failure detection and re-install 59 | -------------------------------------------------------------------------------- /rpi/rpi_detector.py: -------------------------------------------------------------------------------- 1 | import subprocess 2 | import os 3 | import platform 4 | import re 5 | 6 | 7 | HOST_VARS = {} 8 | ANSIBLE_INV = {} 9 | rpi_ip_list = [] 10 | rpi_name_list = [] 11 | 12 | def nmap(): 13 | t = subprocess.Popen("nmap -sP 10.0.0.0/24",shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) 14 | t.wait() 15 | 16 | def pi_search(): 17 | #print ('Searching for RPi') 18 | #print ("---------------------------") 19 | p = subprocess.Popen("arp -a | cut -f 2,4 -d ' ' ", shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) 20 | for line in p.stdout.readlines(): 21 | if line[-18:].startswith('b8:27:eb'):#eth0 mac : b8:27:eb 22 | ip_is = str(re.findall( r'[0-9]+(?:\.[0-9]+){3}',line))[2:-2] 23 | rpi_ip_list.append(ip_is) 24 | #print ("This is RPi IP: " + ip_is) 25 | #print ("---------------------------") 26 | def var_gen_host(): 27 | for i in range(len(rpi_ip_list)): 28 | rpi_name_list.append("rpi"+str(i)) 29 | HOST_VARS[rpi_name_list[i]] = {} 30 | for k in range(len(rpi_name_list)): 31 | HOST_VARS[rpi_name_list[i]]['ansible_ssh_host']=rpi_ip_list[k] 32 | 33 | return (HOST_VARS) 34 | 35 | def var_gen_inv(): 36 | ANSIBLE_INV ["rpi"]={} 37 | ANSIBLE_INV ["rpi"]["hosts"]=rpi_name_list 38 | ANSIBLE_INV ["rpi"]["vars"] = { 39 | "ansible_ssh_user": "root", 40 | "ansible_ssh_private_key_file":"~/.ssh/dd_wrt" 41 | } 42 | return (ANSIBLE_INV) 43 | 44 | 45 | def var_gen(): 46 | var_gen_host() 47 | var_gen_inv() 48 | 49 | 50 | def run(): 51 | pi_search() 52 | 53 | if rpi_ip_list == []: 54 | nmap() 55 | pi_search() 56 | 57 | def main(): 58 | pi_search() 59 | 60 | if rpi_ip_list == []: 61 | print ('Runing nmap') 62 | nmap() 63 | pi_search() 64 | 65 | var_gen() 66 | print ("HOST_VARS") 67 | print (HOST_VARS) 68 | print ("ANSIBLE_INV") 69 | print (ANSIBLE_INV) 70 | #return (HOST_VARS,ANSIBLE_INV) 71 | 72 | 73 | if __name__ == "__main__": 74 | main() 75 | -------------------------------------------------------------------------------- /rpi/rpi_detector.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TranceCat/Raspberry-Pi-orchestration/f6f6e8be0cab6ce55b906574c91fafaf5f0f30ec/rpi/rpi_detector.pyc -------------------------------------------------------------------------------- /rpi/rpi_ssh.ssh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TranceCat/Raspberry-Pi-orchestration/f6f6e8be0cab6ce55b906574c91fafaf5f0f30ec/rpi/rpi_ssh.ssh --------------------------------------------------------------------------------