├── .gitignore
├── README.md
├── monitor.sh
└── support
├── README.md
├── argv
├── btle
├── data
├── init
├── log
├── mqtt
└── time
/.gitignore:
--------------------------------------------------------------------------------
1 | # mac related files
2 | .DS_Store
3 |
4 | # configuration files
5 | address_blacklist
6 | behavior_preferences
7 | known_beacon_addresses
8 | known_static_addresses
9 | mqtt_preferences
10 |
11 | # caches/temp files
12 | .pids
13 | .previous_version
14 | .public_name_cache
15 | .manufacturer_cache
16 |
17 | # Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio and WebStorm
18 | # Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839
19 |
20 | # User-specific stuff
21 | .idea/**/workspace.xml
22 | .idea/**/tasks.xml
23 | .idea/**/usage.statistics.xml
24 | .idea/**/dictionaries
25 | .idea/**/shelf
26 |
27 | # Generated files
28 | .idea/**/contentModel.xml
29 |
30 | # Sensitive or high-churn files
31 | .idea/**/dataSources/
32 | .idea/**/dataSources.ids
33 | .idea/**/dataSources.local.xml
34 | .idea/**/sqlDataSources.xml
35 | .idea/**/dynamic.xml
36 | .idea/**/uiDesigner.xml
37 | .idea/**/dbnavigator.xml
38 |
39 | # Gradle
40 | .idea/**/gradle.xml
41 | .idea/**/libraries
42 |
43 | # Gradle and Maven with auto-import
44 | # When using Gradle or Maven with auto-import, you should exclude module files,
45 | # since they will be recreated, and may cause churn. Uncomment if using
46 | # auto-import.
47 | # .idea/artifacts
48 | # .idea/compiler.xml
49 | # .idea/jarRepositories.xml
50 | # .idea/modules.xml
51 | # .idea/*.iml
52 | # .idea/modules
53 | # *.iml
54 | # *.ipr
55 |
56 | # CMake
57 | cmake-build-*/
58 |
59 | # Mongo Explorer plugin
60 | .idea/**/mongoSettings.xml
61 |
62 | # File-based project format
63 | *.iws
64 |
65 | # IntelliJ
66 | out/
67 |
68 | # mpeltonen/sbt-idea plugin
69 | .idea_modules/
70 |
71 | # JIRA plugin
72 | atlassian-ide-plugin.xml
73 |
74 | # Cursive Clojure plugin
75 | .idea/replstate.xml
76 |
77 | # Crashlytics plugin (for Android Studio and IntelliJ)
78 | com_crashlytics_export_strings.xml
79 | crashlytics.properties
80 | crashlytics-build.properties
81 | fabric.properties
82 |
83 | # Editor-based Rest Client
84 | .idea/httpRequests
85 |
86 | # Android studio 3.1+ serialized cache file
87 | .idea/caches/build_file_checksums.ser
88 |
89 | # Vim related
90 | # Swap
91 | [._]*.s[a-v][a-z]
92 | !*.svg # comment out if you don't need vector files
93 | [._]*.sw[a-p]
94 | [._]s[a-rt-v][a-z]
95 | [._]ss[a-gi-z]
96 | [._]sw[a-p]
97 |
98 | # Session
99 | Session.vim
100 | Sessionx.vim
101 |
102 | # Temporary
103 | .netrwhist
104 | *~
105 | # Auto-generated tag files
106 | tags
107 | # Persistent undo
108 | [._]*.un~
109 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | `monitor`
2 | =======
3 | ***TL;DR***: Passive Bluetooth presence detection of beacons, cell phones, and other Bluetooth devices. Useful for [mqtt-based](http://mqtt.org) home automation, especially when the script runs on multiple devices, distributed throughout a property.
4 |
5 |  
6 |
7 | [**Frequently Asked Questions**](https://github.com/andrewjfreyer/monitor/blob/master/support/README.md)
8 |
9 | Installation Instructions
10 |
11 |
12 |
13 | Set Up Raspberry Pi From Scratch
14 |
15 |
16 | # Installation Instructions for Raspberry Pi Zero W
17 |
18 | ## Setup of SD Card
19 |
20 | 1. Download latest version of **raspbian** [here](https://downloads.raspberrypi.org/raspbian_lite_latest)
21 |
22 | 2. Download etcher from [etcher.io](https://etcher.io)
23 |
24 | 3. Image **raspbian lite buster** to SD card. [Instructions here.](https://www.raspberrypi.org/magpi/pi-sd-etcher/)
25 |
26 | 4. Mount **boot** partition of imaged SD card (unplug it and plug it back in)
27 |
28 | 5. **To enable ssh,** create blank file, without any extension, in the root directory called **ssh**
29 |
30 | 6. **To setup Wi-Fi**, create **wpa_supplicant.conf** file in root directory and add Wi-Fi details for home Wi-Fi:
31 |
32 | ```bash
33 | country=US
34 | ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
35 | update_config=1
36 |
37 | network={
38 | ssid="Your Network Name"
39 | psk="Your Network Password"
40 | key_mgmt=WPA-PSK
41 | }
42 | ```
43 |
44 | 7. **On the first startup,** insert SD card and power on Raspberry Pi Zero W. On first boot, the newly-created **wpa_supplicant.conf** file and **ssh** will be moved to appropriate directories. Find the IP address of the Pi via your router.
45 |
46 | ## Configuration and Setup
47 |
48 | 1. SSH into the Raspberry Pi (default password: raspberry):
49 | ```bash
50 | ssh pi@theipaddress
51 | ```
52 |
53 | 2. Change the default password:
54 | ```bash
55 | sudo passwd pi
56 | ```
57 |
58 | 3. Update and upgrade:
59 |
60 | ```bash
61 | sudo apt-get update
62 | sudo apt-get upgrade -y
63 | sudo apt-get dist-upgrade -y
64 | sudo reboot
65 | ```
66 |
67 | 5. Install Bluetooth Firmware, if necessary:
68 | ```bash
69 | #install Bluetooth drivers for Pi Zero W
70 | sudo apt-get install pi-bluetooth
71 |
72 | ```
73 |
74 | 6. Reboot:
75 | ```bash
76 | sudo reboot
77 | ```
78 |
79 | 7. Install Mosquitto 1.5+ **(important step!)**:
80 | ```bash
81 |
82 | # get repo key
83 | wget http://repo.mosquitto.org/debian/mosquitto-repo.gpg.key
84 |
85 | #add repo
86 | sudo apt-key add mosquitto-repo.gpg.key
87 |
88 | #download appropriate lists file
89 | cd /etc/apt/sources.list.d/
90 | sudo wget http://repo.mosquitto.org/debian/mosquitto-buster.list
91 |
92 | #update caches and install
93 | sudo apt-cache search mosquitto
94 | sudo apt-get update
95 | sudo apt-get install -f libmosquitto-dev mosquitto mosquitto-clients libmosquitto1
96 | ```
97 | Monitor Setup
100 |
101 | ## Setup `monitor`
102 |
103 | 1. Clone `monitor` git:
104 | ```bash
105 | #install git
106 | cd ~
107 | sudo apt-get install git
108 |
109 | #clone this repo
110 | git clone https://github.com/andrewjfreyer/monitor.git
111 |
112 | #enter `monitor` directory
113 | cd monitor/
114 |
115 | #(optional) switch to beta branch for latest updates and features (may be unstable)
116 | git checkout beta
117 |
118 | ```
119 |
120 | 2. Initial run:
121 |
122 | Configuration files will be created with default preferences. Any executables that are not installed will be reported. All can be installed via `apt-get install ...`
123 |
124 | ```bash
125 | sudo bash monitor.sh
126 | ```
127 |
128 |
129 | 3. Edit **mqtt_preferences** file:
130 |
131 | ```bash
132 | sudo nano mqtt_preferences
133 | ```
134 |
135 | 4. Edit **known_static_addresses** (phones, laptops, some smart watches):
136 |
137 | ```bash
138 | sudo nano known_static_addresses
139 | ```
140 |
141 | Alternatively, send an mqtt message to `monitor/setup/ADD STATIC DEVICE` with a message including a mac address and an alias separated by a space:
142 |
143 |
144 | **topic:** `monitor/setup/ADD STATIC DEVICE`
145 | **message:** 00:11:22:33:44:55 alias
146 |
147 |
148 | Use, `monitor/setup/DELETE STATIC DEVICE` with a message containing a mac address to remove a device from all `monitor` nodes.
149 |
150 | 5. Read helpfile:
151 |
152 | ```bash
153 | sudo bash monitor.sh -h
154 | ```
155 |
156 | Now the basic setup is complete. Your broker should be receiving messages and the `monitor` service will restart each time the Raspberry Pi boots. As currently configured, you should run `sudo bash monitor.sh` a few times from your command line to get a sense of how the script works.
157 |
158 | Background & Technical Details
165 |
166 | # *Highlights*
167 |
168 | `monitor` sends a JSON-formatted MQTT message including a confidence value from 0 to 100 to a specified broker when a specified Bluetooth device responds to a `name` query. By default, `name` queries are triggered after receiving an anonymous advertisement from a previously-unseen device (e.g., a device in peripheral mode advertising an ability to connect).
169 |
170 | Example JSON package:
171 | ```
172 | topic: monitor/{{name of monitor install}}/{{mac address}}
173 | message: {
174 | "id":"{{mac address}}",
175 | "confidence":"{{ranging from 0-100}}",
176 | "name":"{{if available}}",
177 | "manufacturer":{{if available}}",
178 | "type":"KNOWN_MAC",
179 | "retained":"{{message retained?}}",
180 | "timestamp":"{{formatted date at which message is sent}}",
181 | "version":"{{monitor version}}"
182 | }
183 | ```
184 |
185 | In addition, optionally, a JSON-formatted MQTT message can be reported to the same broker whenever a publicly-advertising beacon device or an iBeacon device advertises.
186 |
187 | Example JSON package:
188 | ```
189 | topic: monitor/{{name of monitor install}}/{{mac address or ibeacon uuid}}
190 | message: {
191 | "id":"{{mac address or ibeacon uuid}}",
192 | "report_delay":"{{delay from first detection to this message in seconds}}",
193 | "flags":"{{GAP flags}}",
194 | "movement":"stationary",
195 | "confidence":"{{ranging from 0-100}}",
196 | "name":"{{if available}}",
197 | "power":"{{if available}}",
198 | "rssi":"{{if available}}",
199 | "mac":"{{if ibeacon, the current mac address associated with the uuid}}",
200 | "manufacturer":{{if available}}",
201 | "type":"{{GENERIC_BEACON_PUBLIC or APPLE_IBEACON}},
202 | "retained":"{{message retained?}}",
203 | "timestamp":"{{formatted date at which message is sent}}",
204 | "version":"{{monitor version}}"
205 | }
206 | ```
207 | ___
208 |
209 | # *Oversimplified Analogy of the Bluetooth Presence Problem*
210 |
211 | Imagine you're blindfolded in a large room with other people. We want to find out who of your friends **is** present and who of your friends **isn't** present:
212 |
213 | 
214 |
215 | Some of the people in the room periodically make sounds (e.g., eating a chip, sneeze, cough, etc.), others sit quietly and don’t make a sound unless you specifically ask for them by name, and still others periodically announce their own name out loud at regular intervals whether or not you want them to do that:
216 |
217 | 
218 |
219 | Here's the problem. You can’t just shout “WHO’S HERE” because then everyone would say their name at the same time and you couldn’t tell anything apart. Similarly, for obvious reasons, you can't simply ask "WHO ISN'T HERE?"
220 |
221 | So, you take attendance like in a classroom. Everyone in the room responds **only** when their own name is shouted.
222 |
223 | 
224 |
225 | So, one way to take attendance is to shout for each friend on a list by name, one at a time, repeatedly. Ask for someone, get a response, wait for a moment, and ask again.
226 |
227 | Once a friend stops responding (for some period of time), you presume that he or she has left:
228 |
229 | 
230 |
231 | This technique should work just fine, but there's a minor problem. You're constantly shouting into the room, which means that it's difficult for you to hear quiet responses and it's difficult for other people to carry on conversations. What else can we do? Can we use those random sounds for anything?
232 |
233 | Yes! A smarter approach is to wait for an anonymous sound, *then* start asking whether a friend *you know isn't present* has just arrived:
234 |
235 | 
236 |
237 | This way, you're not constantly asking the room for all of your friends. Efficient!
238 |
239 | This technique is a very simplified description of how `monitor` works for devices like cell phones (friends on a list) and beacons (announce a name out loud). This also gives an idea of how `monitor` uses anonymous sounds to reduce the number of times that it has to send inquiries into the Bluetooth environment.
240 |
241 | ___
242 |
243 | # *Oversimplified Technical Background*
244 |
245 | The Bluetooth Low Energy spec was designed to make connecting Bluetooth devices simpler for the user. No more pin codes, no more code verifications, no more “discovery mode” - for the most part. It was also designed to be more private than previous Bluetooth implementations. That said, it’s hard to maintain privacy when you want to be able to connect to an unknown device without intervention.
246 |
247 | ## Name Requests
248 |
249 | A part of the Blueooth spec is a special function called a `name` request that asks another Bluetooth device to send back a human-readable name of itself. In order to send a `name` request, however, we need to know a private (unchanging) address of the target device.
250 |
251 | Issuing a `name` request to the same private mac address every few seconds is a reliable - albeit rudimentary - way of detecting whether that device is "**present**" (it responds to the `name` request) or "**absent**" (no response to the `name` request is received). However, issuing `name` requests too frequently (*e.g.*, every few seconds) uses quite a bit of 2.4GHz spectrum, which can cause interference with Wi-Fi or other wireless communications.
252 |
253 | Not all devices respond to `name` requests, however. For example, beacon devices do not respond.
254 |
255 | ## Connectible Devices
256 |
257 | Blueooth devices that can exchange information with other devices (almost always) advertise a random/anonymous address that other devices can use to negotiate a secure connection and receive the first device's real, private, Bluetooth address. Using a random address in this way when publicly advertising prevents bad actors from tracking via passive Bluetooth monitoring.
258 |
259 | ## Beacon/Advertising Devices
260 |
261 | The Bluetooth spec has been used by Apple, Google, and others to create additional standards (e.g., iBeacon, Eddystone, and so on). These devices generally don't care to connect to other devices, so use of random/anonymous addresses doesn't really matter. Instead, these devices encode additional information into each advertisement of an anonymous address. For example, iBeacon devices will broadcast a UUID that conforms to the 8-4-4-4-12 format defined by [IETC RFC4122](http://www.ietf.org/rfc/rfc4122.txt).
262 |
263 | As noted above, most beacons do not respond to `name` requests, even if made to the device's private Bluetooth address. So, issuing periodic `name` requests to beacons is not a good way to detect whether a beacon device is **present** or **absent**. However, monitoring for beacon advertisement is a reliable way to detect whether a beacon device is **present** or **absent**.
264 |
265 | _____
266 |
267 | # *How `monitor` Works*
268 |
269 | This script combines `name` requests, anonymous advertisements, and beacon advertisements to logically determine (1) *when* to issue a `name` request to determine whether a device is **present** and (2) *when* to issue a `name` request to determine whether a device is **absent**. The script also listens for beacons.
270 |
271 | ##### Known Static Addresses
272 | `monitor` uses unchanging/static mac addresses for your devices that you have added to a file called `known_static_addresses`. These are the addresses for which `monitor` will issue `name` requests to determine whether or not these devices are **present** or **absent**.
273 |
274 | Once a determination of presence is made, the script posts to an mqtt topic path defined in a file called `mqtt_preferences` that includes a JSON-formatted message with a confidence value that corresponds to a confidence of presence. For example, a confidence of 100 means that `monitor` is 100% sure the device is present. Similarly, a confidence of 0 means that `monitor` is 0% sure the device is present (*i.e.*, the `monitor` is 100% sure the device is absent).
275 |
276 | To minimize the number of times that `monitor` issues `name` requests (thereby reducing 2.4GHz interference), the script performs either an ***ARRIVAL*** scan or a ***DEPART*** scan, instead of scanning all devices listed in the `known_static_addresses` each time.
277 |
278 | More specifically:
279 |
280 | * An ***ARRIVAL*** scan issues a `name` request, sequentially, for each device listed in the `known_static_addresses` file that is known to be **absent**.
281 |
282 | * Similarly, a ***DEPART*** scan issues a `name` request, sequentially, for each device listed in the `known_static_addresses` file that is known to be **present**.
283 |
284 | For example, if there are two phone addresses listed in the `known_static_addresses` file, and both of those devices are **present**, an ***ARRIVAL*** scan will never occur. Similarly, if both of these addresses are **absent** then a ***DEPART*** scan will never occur. If only one device is present, an **ARRIVAL** scan will only scan for the device that is currently away.
285 |
286 | To reduce the number of `name` requests that occur, `monitor` listens for anonymous advertisements and triggers an ***ARRIVAL*** scan for every *new* anonymous address.
287 |
288 | The script will also trigger an ***ARRIVE*** scan in response to an mqtt message posted to the topic of `monitor/scan/arrive`. Advertisement-triggered scanning can be disabled by using the trigger argument if `-ta`, which causes `monitor` to *only* trigger ***ARRIVAL*** scans in response to mqtt messages.
289 |
290 | If `monitor` has not heard from a particular anonymous address in a long time, `monitor` triggers a ***DEPART*** scan. The script will also trigger a ***DEPART*** scan in response to an mqtt message posted to the topic of `monitor/scan/depart`. Expiration-triggered scanning can be disabled by using the trigger argument if `-td`, which causes `monitor` to *only* trigger ***DEPART*** scans in response to mqtt messages.
291 |
292 | To reduce scanning even further, `monitor` can filter which types of anonymous advertisements are used for ***ARRIVE*** scans. These are called "filters" and are defined in a file called `behavior_preferences`. The filters are bash RegEx strings that either pass or reject anonymous advertisements that match the filter.
293 |
294 | There are two filter types:
295 |
296 | * **Manufacturer Filter** - filters based on data in an advertisement that is connected to a particular device manufacturer. This is almost always the OEM of the device that is transmitting the anonymous advertisement. By default, because of the prevalence of iPhones, Apple is the only manufacturer that triggers an ***ARRIVAL*** scan. Multiple manufacturers can be appended together by a pipe: `|`. An example filter for Apple and Samsung looks like: `Apple|Samsung`. To disable the manufacturer filter, use `.*`.
297 |
298 | * **Flag Filter:** filters based on flags contained in an advertisement. This varies by device type. By default, because of the prevalence of iPhones, the flag of `0x1b` triggers an ***ARRIVAL*** scan. Like with the manufacturer filter, multiple flags can be appended together by a pipe: `|`. To disable the manufacturer filter, use `.*`.
299 |
300 | ##### Beacons & iBeacons
301 | In addition, when run with the `-b` beacon argument, `monitor` listens for beacon advertisements that report themselves as "public", meaning that their addresses will not change. The script can track these by default; these addresses do not have to be added anywhere - after all, `monitor` will obtain them just by listening.
302 |
303 | Since iBeacons include a UUID and a mac address, two presence messages are reported via mqtt.
304 |
305 | ## Known Beacon Addresses
306 | In some cases, manufacturers try to get sneaky and cause their beacons to advertise as "anonymous" (or "random") devices, despite that their addresses do not change at all. By default, `monitor` does not report presence of anonymous advertisement devices, so to force `monitor` to recognize these devices, we add the "random" address to a file called `known_static_beacons`. After restarting, `monitor` will know that these addresses should be treated like a normal beacon.
307 | ___
308 |
309 | Home Assistant Example
312 |
313 | # Example with Home Assistant
314 |
315 | Personally, I have four **raspberry pi zero w**s throughout the house and garage. My family spends most of our time on the first floor, so our main `monitor` node or sensor is on the first floor. Our other 'nodes' on the second and third floor and garage are set up for triggered use only - these will scan for ***ARRIVAL*** and ***DEPART*** only in response to mqtt messages, with option ```-tad```. The first floor node is set up to send mqtt arrive/depart scan instructions to these nodes by including the `-tr` flag ("report" to other nodes when an arrival or depart scan is triggered).
316 |
317 | The first floor constantly monitors for beacons (`-b`) advertisements and anonymous advertisements, which may be sent by our phones listed in the `known_static_addresses` file. In response to a new anonymous advertisement, `monitor` will initiate an ***ARRIVAL*** scan for whichever of our phones is not present. If one of those devices is seen, an mqtt message is sent to Home Assistant reporting that the scanned phone is "home" with a confidence of 100%. In addition, an mqtt message is sent to the second and third floor and garage to trigger a scan on those floors as well. As a result of this configuration, when we leave the house, we use either the front door or the garage door to trigger an mqtt trigger of ```monitor/scan/depart``` after a ten second delay to trigger a departure scan of our devices that were previously known to be present. The ten second delay gives us a chance to get out of Bluetooth range before a "departure" scan is triggered. Different houses/apartments will probably need different delays.
318 |
319 | More specifically, each of these `monitor` nodes uses the same name for each device so that states can be tracked easily by Home Assistant. For example, on each node, my `known_static_addresses` file looks like this (note that 00:00:00:00:00:00 is an example address - this should be your phone's private, static, Bluetooth address):
320 |
321 | ```bash
322 | 00:00:00:00:00:00 alias #comment that is ignored
323 | ```
324 |
325 | The address I want to track is separated by a space from the *alias* that I want to use to refer to this device in Home Assistant. If you prefer to use the address instead of an alias, set the value `PREF_ALIAS_MODE=false` in your `behavior_preferences` file.
326 |
327 | In this manner, [Home Assistant](https://www.home-assistant.io) receives mqtt messages and stores the values as input to a number of [mqtt sensors](https://www.home-assistant.io/components/sensor.mqtt/). Output from these sensors is combined to give an accurate numerical occupancy confidence:
328 |
329 | ```
330 | - platform: mqtt
331 | state_topic: 'monitor/first floor/alias'
332 | value_template: '{{ value_json.confidence }}'
333 | unit_of_measurement: '%'
334 | name: 'First Floor'
335 |
336 | - platform: mqtt
337 | state_topic: 'monitor/second floor/alias'
338 | value_template: '{{ value_json.confidence }}'
339 | unit_of_measurement: '%'
340 | name: 'Second Floor'
341 |
342 | - platform: mqtt
343 | state_topic: 'monitor/third floor/alias'
344 | value_template: '{{ value_json.confidence }}'
345 | unit_of_measurement: '%'
346 | name: 'Third Floor'
347 |
348 | - platform: mqtt
349 | state_topic: 'monitor/garage/alias'
350 | value_template: '{{ value_json.confidence }}'
351 | unit_of_measurement: '%'
352 | name: 'Garage'
353 | ```
354 |
355 | These sensors can be combined using a [min_max](https://www.home-assistant.io/components/sensor.min_max/):
356 |
357 | ```
358 | - platform: min_max
359 | name: "Home Occupancy Confidence"
360 | type: max
361 | round_digits: 0
362 | entity_ids:
363 | - sensor.third_floor
364 | - sensor.second_floor
365 | - sensor.first_floor
366 | - sensor.garage
367 | ```
368 |
369 | Thereafter, I use the entity **sensor.home_occupancy_confidence** in automations to control the state of an **input_boolean** that represents a very high confidence of a user being home or not.
370 |
371 | As an example:
372 |
373 | ```
374 | - alias: Occupancy On
375 | trigger:
376 | - platform: numeric_state
377 | entity_id: sensor.home_occupancy_confidence
378 | above: 10
379 | action:
380 | - service: input_boolean.turn_on
381 | data:
382 | entity_id: input_boolean.occupancy
383 |
384 | - alias: Occupancy Off
385 | trigger:
386 | - platform: numeric_state
387 | entity_id: sensor.home_occupancy_confidence
388 | below: 10
389 | action:
390 | - service: input_boolean.turn_off
391 | data:
392 | entity_id: input_boolean.occupancy
393 | ```
394 |
395 | If you prefer to use the `device_tracker` platform in Home Assistant, a unique solution is to use the undocumented `device_tracker.see` service:
396 |
397 | As an example:
398 |
399 | ```
400 | - alias: Andrew Occupancy On
401 | trigger:
402 | - platform: numeric_state
403 | entity_id: sensor.andrew_occupancy_confidence
404 | above: 10
405 | action:
406 | - service: device_tracker.see
407 | data:
408 | dev_id: andrew
409 | location_name: home
410 | source_type: bluetooth
411 |
412 | - alias: Andrew Occupancy Off
413 | trigger:
414 | - platform: numeric_state
415 | entity_id: sensor.andrew_occupancy_confidence
416 | below: 10
417 | action:
418 | - service: device_tracker.see
419 | data:
420 | dev_id: andrew
421 | location_name: not_home
422 | source_type: bluetooth
423 |
424 | ```
425 |
426 | For more information, see [here](https://community.home-assistant.io/t/device-tracker-from-script/97295/7) and [here](https://github.com/andrewjfreyer/monitor/issues/138).
427 |
428 | If you only have one node, an [add-on](https://github.com/Limych/hassio-addons) by @limych may be an excellent choice for you!
429 |
430 | If having several nodes and lots of users, there is an AppDaemon App that can easily manage the integration of of this system into Home Assistant automatically for you by @Odianosen25. More information can be found [here](https://github.com/Odianosen25/Monitor-App)
431 |
432 | Advanced Configuration Options & Fine Tuning
435 |
436 |
437 | ## Fine Tuning
438 |
439 |
440 | 1. Observe output from `monitor` to tune filters:
441 |
442 | ```bash
443 | sudo bash monitor.sh
444 | ```
445 |
446 | Observe the output of the script for debug log [CMD-RAND] lines including [failed filter] or [passed filter]. These lines show what anonymous advertisement `monitor` sees and how `monitor` filters those advertisements. In particular, cycle the Bluetooth power on your phone or another device and look at the `flags` value, the `pdu` value, and the `man` (manufacturer) value that appears after you turn Bluetooth power back on. Remember, the address you see in the log will be an anonymous address - ignore it, we're only focused on the values referenced above.
447 |
448 | ```
449 | 0.1.xxx 03:25:39 pm [CMD-RAND] [passed filter] data: 00:11:22:33:44:55 pdu: ADV_NONCONN_IND rssi: -73 dBm flags: 0x1b man: Apple, Inc. delay: 4
450 | ```
451 |
452 | If you repeatedly see the same values in one or more of these fields, consider adding a PASS filter condition to the `behavior_preferences` file. This will cause `monitor` to *only* scan in response to an anonymous advertisement that passes the filter condition that you define. For example, if you notice that Apple always shows up as the manufacturer when you cycle the power on you phone, you can create an Apple filter:
453 |
454 | ```bash
455 | PREF_PASS_FILTER_MANUFACTURER_ARRIVE="Apple"
456 | ```
457 |
458 | If you have two phones, and one is **Apple** and the other is **Google**, create a `bash` or statement in the filter like this:
459 |
460 | ```bash
461 | PREF_PASS_FILTER_MANUFACTURER_ARRIVE="Apple|Google"
462 | ```
463 |
464 | If your phone shows as **Unknown**, then it is best to disable the filter entirely - some phones will report a blank manufacturer, others will report a null value... it's much easier to try and filter with another value:
465 |
466 | ```bash
467 | PREF_PASS_FILTER_MANUFACTURER_ARRIVE=".*"
468 | ```
469 |
470 | Similarly, we can create a negative filter. If you or your neighbors use Google Home, it is likely that you'll see at least some devices manufactured by **Google**. Create a fail filter condition to ignore these advertisements:
471 |
472 | ```bash
473 | PREF_FAIL_FILTER_MANUFACTURER_ARRIVE="Google"
474 | ```
475 |
476 | Filters are a great way to minimize the frequency of `name` requestning, which causes 2.4GHz interference and can, if your values are too aggressive, dramatically interfere with Wi-Fi and other services.
477 |
478 | 2. **Standard configuration options:**
479 |
480 | When `monitor` is first run, default preferences are created in the `behavior_preferences` file. These preferences can be changed, and in many cases should be changed depending on your Bluetooth environment (how many devices you have around you at any given time). A table below describes what these default variables are:
481 |
482 | | **Option** | **Default Value** | **Description** |
483 | |-|-|-|
484 | | PREF_ARRIVAL_SCAN_ATTEMPTS | 1 | This is the number of times that `monitor` will send a name request before deciding that a device has not yet arrived. The higher the number, the fewer errors on arrival detection but also the longer it may take to recognize all devices are home in a multi-device installation. |
485 | | PREF_DEPART_SCAN_ATTEMPTS | 2 | This is the number of times that `monitor` will send a name request before deciding that a device has not yet departed. The higher the number, the fewer errors on departure detection but also the longer it may take to recognize all devices are away in a multi-device installation. |
486 | | PREF_BEACON_EXPIRATION | 180 | This is the number of seconds without observing an advertisement before a beacon is considered expired. |
487 | | PREF_MINIMUM_TIME_BETWEEN_SCANS | 15 | This is the minimum number of seconds required between "arrival" scans or between "departure" scans. Increasing the value will decrease interference, but will also increase arrival and departure detection time. |
488 | | PREF_PASS_FILTER_ADV_FLAGS_ARRIVE | .* | See above. |
489 | | PREF_PASS_FILTER_MANUFACTURER_ARRIVE | .* | See above. |
490 | | PREF_FAIL_FILTER_ADV_FLAGS_ARRIVE | NONE | See above. |
491 | | PREF_FAIL_FILTER_MANUFACTURER_ARRIVE | NONE | See above. |
492 | | PREF_ALIAS_MODE | true | Disable or enable alias mode; if disabled, MQTT messages are sent using a device's mac address. |
493 |
494 | 3. **Advanced configuration options:**
495 |
496 | In addition to the options described above, there are a number of advanced options that can be set by the user. To modify any of these options, add a line to the `behavior_preferences` file.
497 |
498 |
499 | | **Option** | **Default Value** | **Description** |
500 | |-|-|-|
501 | PREF_INTERSCAN_DELAY|3|This is a fixed delay between `name` requests. Increasing the value will decrease interference, but will decrease responsiveness. Decreasing the value will risk a Bluetooth hardware fault.|
502 | PREF_RANDOM_DEVICE_EXPIRATION_INTERVAL|75|This is the interval after which an anonymous advertisement mac address is considered expired. Increasing this value will reduce arrival scan frequency, but will also increase memory footprint (minimal) and will decrease the frequency of depart scans.|
503 | PREF_RSSI_CHANGE_THRESHOLD|-20|If a beacon's rssi changes by at least this value, then the beacon will be reported again via mqtt.|
504 | PREF_RSSI_IGNORE_BELOW|-75|If an anonymous advertisement is "farther" away (lower RSSI), ignore the advertisement
505 | PREF_HCI_DEVICE|hci0|Select which hci device should be used by `monitor`|
506 | PREF_COOPERATIVE_SCAN_THRESHOLD|60|Once confidence of a known device falls below this value, send an mqtt message to other `monitor` nodes to begin an arrival scan or a departure scan.|
507 | PREF_MQTT_REPORT_SCAN_MESSAGES|false|This value is either true or false and determines whether `monitor` publishes when a scan begins and when a scan ends|
508 | PREF_PERCENT_CONFIDENCE_REPORT_THRESHOLD|59|This value defines when a beacon begins reporting a decline in confidence|
509 | PREF_PASS_FILTER_PDU_TYPE|*Various. See FAQ.*|These are the PDU types that should be noticed by `monitor`|
510 | PREF_DEVICE_TRACKER_REPORT|false|If true, this value will cause `monitor` to report a 'home' or 'not_home' message to `... /device_tracker` conforming to device_tracker mqtt protocol.
511 | PREF_DEVICE_TRACKER_HOME_STRING|home|If `PREF_DEVICE_TRACKER_REPORT` is true, this is the string that is reported to the device_tracker when the device is home.
512 | PREF_DEVICE_TRACKER_AWAY_STRING|not_home|If `PREF_DEVICE_TRACKER_REPORT` is true, this is the string that is reported to the device_tracker when the device is not home.
513 | PREF_DEVICE_TRACKER_TOPIC_BRANCH|device_tracker|If `PREF_DEVICE_TRACKER_REPORT` is true, this is last path element of the mqtt topic path that will be used to publish the device tracker message.
514 | PREF_ADVERTISEMENT_OBSERVED_INTERVAL_STEP|15|This is the minimum interval (in seconds) used to estimate advertisement intervals reported in the MQTT message.
515 | PREF_DEPART_SCAN_INTERVAL|30|If using periodic scanning mode, this is the minimum interval (in seconds) at which depart scans are triggered automatically.
516 | PREF_ARRIVE_SCAN_INTERVAL|15|If using periodic scanning mode, this is the minimum interval (in seconds) at which arrive scans are triggered automatically.
517 |
518 |
519 | ## RSSI Tracking
520 |
521 | This script can also track RSSI changes throughout the day. This can be used for very rudimentary room- or floor-level tracking. Only devices in `known_static_addresses` that have been paired to a `monitor` node can have their RSSI tracked. Here's how to pair:
522 |
523 | 1. Stop `monitor` service:
524 |
525 | ```bash
526 | sudo systemctl stop monitor
527 | ```
528 |
529 | 2. Run `monitor` with `-c` flag, followed by the mac address of the known_device to connect:
530 |
531 | ```bash
532 | sudo bash monitor.sh -c 00:11:22:33:44:55
533 | ```
534 |
535 | After this, follow the prompts given by `monitor` and your device will be connected. That's it. After you restart monitor will periodically (once every ~1.5 minutes) connect to your phone and take three RSSI samples, average the samples, and report a string message to the same path as a confidence report, with the additional path component of */rssi*. So, if a `monitor` node is named 'first floor', an rssi message is reported to:
536 |
537 | ```bash
538 | topic: monitor/first floor/00:11:22:33:44:55/rssi
539 | message: -99 through 0
540 | ```
541 |
542 | If an rssi measurement cannot be obtained, the value of -99 is sent.
543 |
544 | ## Report known states
545 |
546 | It is also possible tell monitor to report all currently known device states by sending an MQTT message to something like `monitor/first floor/KNOWN DEVICE STATES`. monitor.sh will then iterate over all known static addresses and report the current confidence level. This may be useful in home assistant to get the current state after a home assistant restart.
547 |
548 | Will this be able to track my Apple Watch/Smart Watch?
4 |
5 | Yes, with a caveat. Many users, including myself, have successfully added Apple Watch Bluetooth addresses to the `known_static_addresses` file. In my personal experience, an Apple Watch works just fine [once it has connected to at least one other Bluetooth device, apart from your iPhone](https://github.com/andrewjfreyer/monitor#my-phone-doesnt-seem-to-automatically-broadcast-an-anonymous-bluetooth-advertisement-what-can-i-do). Other users have reported that the Apple Watch will occasionally not respond to this script. Your mileage using the Apple Watch and/or other low-power connectible Bluetooth devices may vary. I strongly recommend tracking phones.
6 |
7 |
What special app do I need on my phone to get this to work?
10 |
11 | None, except in rare circumstances. The only requirement is that Bluetooth is left on. Works best with iPhones and Android phones that have peripheral mode enabled.
12 |
Does this script reduce battery life for my phone?
15 |
16 | Not noticeable in my several years of using techniques similar to this.
17 |
How can I trigger an arrival scan?
20 |
21 | Post a message with blank content to `monitor/scan/arrive`
22 |
How can I trigger an depart scan?
25 |
26 | Post a message with blank content to `monitor/scan/depart`
27 |
How can I trigger an arrive/depart scan from an automation in Home Assistant?
30 |
31 | For an automation or script (or other service trigger), use:
32 |
33 | ```yaml
34 | service: 'mqtt.publish'
35 | data:
36 | topic: monitor/scan/arrive
37 | ```
38 |
39 | ```yaml
40 | service: 'mqtt.publish'
41 | data:
42 | topic: monitor/scan/depart
43 | ```
44 |
How can I add a known device without manually entering an address?
47 |
48 | Post a message with the mac address separated from an alias (optional) by a space to: `monitor/setup/add known device`
49 |
How can I delete a known device without manually editing an address?
52 |
53 | Post a message with the mac address to: `monitor/setup/delete known device`
54 |
How can I upgrade to the latest version without using ssh?
57 |
58 | Post a message with blank content to `monitor/scan/update` or `monitor/scan/updatebeta`
59 |
How can I restart a this script node?
62 |
63 | Via command line:
64 |
65 | ```bash
66 | sudo systemctl restart monitor
67 | ```
68 |
69 | Or, post a message with blank content to `monitor/scan/restart`
70 |
Why don't I see RSSI for my device?
73 |
74 | For phones, you'll have to connect to `monitor` first using the `-c` flag.
75 |
How do I force an RSSI update for a known device, like my phone?
78 |
79 | Post a message with blank content to `monitor/scan/rssi`
80 |
81 |
Can't I just issue a name scan every few seconds to get faster arrival and depart detection?
88 |
89 | Yes, use periodic scanning mode with `-r`.
90 |
Can I use other Bluetooth services while this script is running?
93 |
94 | No. Monitor needs exclusive use of the Bluetooth radio to function properly. This is why it is designed to run on inexpensive hardware like the Raspberry Pi Zero W.
95 |
Can this script run on XYZ hardware or in XYZ container?
98 |
99 | Probably. The script has been designed to minimize dependencies as much as possible. That said, I can't guarantee or provide support to all systems.
100 |
Does this script interfere with Wi-Fi, Zigbee, or Zwave?
103 |
104 | It can, if it scans too frequently, especially if you're running this script from internal Raspberry Pi radios. Try to use all techniques for reducing `name` scans, including using trigger-only depart mode `-tdr`. When in this mode, this script will never scan when all devices are home. Instead, this script will wait until a `monitor/scan/depart` message is sent. Personally, I use my front door lock as a depart scan trigger.
105 |
How can I check if a this script node is up and hasn't shut down for some reason?
108 |
109 | Post a message to `monitor/scan/echo`, and you'll receive a response at the topic `$mqtt_topicpath/$mqtt_publisher_identity/echo`
110 |
I have interference and/or my ssh sessions are really slow and laggy. What gives?
113 |
114 | Cheap Wi-Fi chipsets and cheap Bluetooth chipsets can perform poorly together if operated at the same time, especially on Raspberry Pi devices. If you still experience interference in your network, switching to a Wi-Fi dongle can help.
115 |
I use a Bluetooth dongle, and this script seems to become non-responsive after a while - what's going on?
118 |
119 | Many Bluetooth dongles do not properly filter out duplicate advertisements, so this script gets overwhelmed trying to filter out hundreds of reports, when it expects dozens. I'm working on a solution, but for now the best option is to switch to internal Bluetooth or, alternatively, you can try another Bluetooth dongle.
120 |
I keep seeing that my Bluetooth hardware is "cycling" in the logs - what does that mean?
127 |
128 | If more than one program or executable try to use the Bluetooth hardware at the same time, your Bluetooth hardware will report an error. To correct this error, the hardware needs to be taken offline, then brought back.
129 |
How do I access logs?
132 |
133 | Run via command line and post log output to github. Else, access `journalctl` to show the most recent logs:
134 |
135 | ```bash
136 | journalctl -u monitor -r
137 | ```
138 |
My Android phone doesn't seem to send any anonymous advertisements, no matter what I do. Is there any solution?
141 |
142 | Some phones, like the LG ThinQ G7 include an option in settings to enable file sharing via bluetooth. As resported by Home Assistant forum user @jusdwy, access this option via Settings >Connected Devices > File Sharing > File Sharing ON. For other android phones, an app like [Beacon Simulator](https://play.google.com/store/apps/details?id=net.alea.beaconsimulator&hl=en_US) may be a good option. You may also be able to see more information about Bluetooth on your phone using [nRF Connect](https://play.google.com/store/apps/details?id=no.nordicsemi.android.mcp&hl=en_US).
143 |
144 | Unfortunately, until Android OS includes at least one service that requires bluetooth peripheral mode to be enabled, Android devices will probably not advertise without an application running in the background. In short, as I understand it, Android/Google has been slow to adopt BTLE peripheral mode as an option in addition to the default central mode. [Here is a decently comprehensive list of phones that support peripheral mode](https://altbeacon.github.io/android-beacon-library/beacon-transmitter-devices.html), should an application choose to leverage the appropriate API. It does not appear as though the native OS has an option (outside of the file sharing option mentioned above on LG phones) to enable this mode.
145 |
146 | Unfortunately, it seems to me that absent an application causing an advertisement to send, Android users will not be able to use monitor in the same way as iOS users or beacon users.
147 |
My phone doesn't seem to automatically broadcast an anonymous Bluetooth advertisement ... what can I do?
150 |
151 | Many phones will only broadcast once they have already connected to *at least one* other Bluetooth device. Connect to a speaker, a car, a headset, or `monitor.sh -c [address]` and try again.
152 |
Why does my MQTT broker show connection and disconnection so often?
155 |
156 | This is normal behavior for `mosquitto_pub` - nothing to worry about.
157 |
I updated and this script is no longer working ... what gives?
160 |
161 | Make sure you've updated `mosquitto` to v1.5 or higher. In order to support a wider userbase, backward compatibility for old versions of `mosquitto` was dropped. It is alos strongly recommended that you upgrade to bash 4.4+.
162 |
I keep seeing MQTT Broker Offline messages in the this script log. What's going on?
165 |
166 | mosquitto fails to connect to a broker if your password has certain special characters such as: `@`, `:`,`/` - if this is the case, the easiest solution is to create a new user for this script with a different password.
167 |
Can I use a certfile for mosquitto instead of my password?
170 |
171 | Yes, specify a path for `mqtt_certificate_path` in mqtt_preferences.
172 |
What filters do you personally use?
179 |
180 | ```bash
181 |
182 | #ARRIVE TRIGGER FILTER(S)
183 | PREF_PASS_FILTER_ADV_FLAGS_ARRIVE=\"0x1a|0x1b\"
184 | PREF_PASS_FILTER_MANUFACTURER_ARRIVE=\"Apple\"
185 |
186 | #ARRIVE TRIGGER NEGATIVE FILTER(S)
187 | PREF_FAIL_FILTER_MANUFACTURER_ARRIVE=\"Google|Samsung\"
188 | PREF_FAIL_FILTER_MANUFACTURER_ARRIVE=\"NONE\"
189 | ```
190 |
What are the default filters for the PDU filter option?
193 |
194 | ```ADV_IND|ADV_SCAN_IND|ADV_NONCONN_IND|SCAN_RSP```
195 |
How do I use this as a device_tracker, in addition to the standard confidence messages?
198 |
199 | Set the option `PREF_DEVICE_TRACKER_REPORT` in your `behavior_preferences` file to true. If it's not there, add a line like this:
200 |
201 | ```bash
202 | PREF_DEVICE_TRACKER_REPORT=true
203 | ```
204 |
205 | Then, an additional mqtt message will be posted to the topic branch ending in `/device_tracker`
206 |
207 | So, as an example for a this script node named "first floor", a device tracker configuration for Home Assistant can look like:
208 |
209 | ```yaml
210 |
211 | device_tracker:
212 | - platform: mqtt
213 | devices:
214 | andrew_first_floor: 'monitor/first floor/[device address or alias]/device_tracker'
215 | ```
216 |
217 | The standard confidence report will also send.
218 |
How do I determine what values to set for filters?
221 |
222 | Try using the verbose logging option `-V` to see what this script sees when a new bluetooth device advertisement is seen. Then, power cycle the bluetooth radio on the device you'd like to track - you'll probably see a pattern develop with flags or manufacturers. Use these values to create your arrival filters!
223 |
224 | Similarly, to set exclude filters, you can observe bluetooth traffic for a period of time to see what devices you simply do not care about seeing.
225 |
It's annoying to have to keep track of mac addresses. Can't I just use a nickname for the mac addresses for MQTT topics?
232 |
233 | Yes, this is default behavior. All you have to do is provide a name next to the address in the `known_static_addresses` file. For example, if you have a known device with the mac address of 00:11:22:33:44:55 that you would like to call "Andrew's Phone":
234 |
235 | ```bash
236 | 00:11:22:33:44:55 Andrew's iPhone
237 | ```
238 |
239 | Then restart the this script service. The script will now use "andrew_s_iphone" as the final mqtt topic path component.
240 |
241 | ***Important:***
242 |
243 | * any entry will be made **lowercase**
244 |
245 | * any non-digit or non-decimal character will be replaced with an underscore
246 |
247 | The same is true for beacons in the `known_beacon_addresses` file as well:
248 |
249 | ```bash
250 | 09876543-3333-2222-1111-000000000000-9-10000 Dog
251 | ```
252 |
253 | To disable this feature, set `PREF_ALIAS_MODE=false` in your `behavior_preferences` file.
254 |
I don't care about a few devices that are reporting. Can I block them?
257 |
258 | Yes. Create a file called `address_blacklist` in your configuration directory and add the mac addresses you'd like to block (or uuid-major-minor for iBeacons) one at a time.
259 |
I can't use the device_tracker platform with the default status strings of `home` and `not_home` with my home automation software. What can I do?
262 |
263 | Set these options in `behavior_preferences`:
264 |
265 | ```bash
266 | PREF_DEVICE_TRACKER_HOME_STRING='home status string'
267 | PREF_DEVICE_TRACKER_AWAY_STRING='away status string'
268 | PREF_DEVICE_TRACKER_TOPIC_BRANCH='topic path for device tracker/presence tracker'
269 | ```
270 |
271 | Examples:
272 |
273 | Home Assistant (default):
274 |
275 | ```bash
276 | PREF_DEVICE_TRACKER_HOME_STRING='home'
277 | PREF_DEVICE_TRACKER_AWAY_STRING='not_home'
278 | PREF_DEVICE_TRACKER_TOPIC_BRANCH='device_tracker'
279 | ```
280 |
281 | SmartThings:
282 |
283 | ```bash
284 | PREF_DEVICE_TRACKER_HOME_STRING='present'
285 | PREF_DEVICE_TRACKER_AWAY_STRING='not present'
286 | PREF_DEVICE_TRACKER_TOPIC_BRANCH='presence'
287 | ```
288 |
289 | Generic:
290 |
291 | ```bash
292 | PREF_DEVICE_TRACKER_HOME_STRING='home'
293 | PREF_DEVICE_TRACKER_AWAY_STRING='away'
294 | PREF_DEVICE_TRACKER_TOPIC_BRANCH='anything you like'
295 | ```
296 |
297 |