├── README.md ├── ais-server.py └── ship-utils.py /README.md: -------------------------------------------------------------------------------- 1 | # Ambient Shipping 2 | 3 | This repo contains utilities for capturing AIS messages broadcast by passing ships and then joining them with public data sets that reveal what the ships are carrying. 4 | 5 | ### Overview: 6 | Ambient Shipping enables you to look inside cargo ships. It has two main components: 7 | 8 | - `ship-utils.py`: A simple set of functions that enables (1) the translation between an MMSI and a Vessel's Name and (2) wrappers around the [Enigma Public API](https://public.enigma.com) to query Bill of Lading Import Records for that Vessel to reveal what is inside it. 9 | 10 | - `ais-server.py`: A data relay server for relaying decoded AIS transmissions to a remote database server. It is intended to be run on a small linux machine with a SDR (e.g. a raspberry pi) and in proximity to waterways frequented by cargo ships. 11 | 12 | ![NY Harbor](https://s3.amazonaws.com/marcdacosta.com/storage/nyharbor.jpg) 13 | 14 | 15 | ### Context: 16 | 17 | The [Automatic Identification System (AIS)](https://en.wikipedia.org/wiki/Automatic_identification_system) is a broadcast radio signal that reports a ship's position, speed, heading and other metadata about its movements. It is primarily used for safety and for ships to have situational awareness. The signals are broadcast on two VHF channels (161.975 MHz, 162.025 MHz) and generally have a range of 10-20 miles at sea, although, on land, buildings interfere with the signal and it can be difficult to receive without a clear line of sight for the antenna. 18 | 19 | ![AIS Transceiver](https://s3.amazonaws.com/marcdacosta.com/storage/Matsutec-boat-GPS-navigation-equipment-5-6-Color-LCD-Marine-GPS-SBAS-Navigator-w-High-Sensitivity.jpg_640x640.jpg) 20 | 21 | 22 | There are many different types of AIS messages that are broadcast ([detailed here](https://en.wikipedia.org/wiki/Automatic_identification_system#AIS_messages)) but, most commonly, an AIS transceiver sends basic positional data every 2 to 10 seconds depending on a vessel's speed while underway, and every 3 minutes while a vessel is at anchor. 23 | 24 | Using a [RTL-SDR](https://www.amazon.com/NooElec-NESDR-Mini-Compatible-Packages/dp/B009U7WZCA) dongle, these transmissions can be picked up and decoded. The [RTL-AIS](https://github.com/dgiardini/rtl-ais) project is quite useful as a local server which will receive AIS messages and output them on a local host port. It must be run as a background process for `ais-server.py` to function properly. 25 | 26 | Example AIS data: 27 | 28 | `!AIVDM,1,1,,B,15MwDp0P0hJe>pLGBUq;q?wN2@KI,0*27` 29 | 30 | `!AIVDM,1,1,,B,15N85>PP00re>T8GBVmf4?v:2<4R,0*6E` 31 | 32 | 33 | The [spec](http://catb.org/gpsd/AIVDM.html#_aivdm_aivdo_sentence_layer) for AIS encoding is quite complicated, but convenient tools like Python's [libais](https://pypi.python.org/pypi/libais) make it straightforward to decode. Using this in conjunction with the `ais-server.py` script will enable received messages to be uploaded to a remote database for later analysis. 34 | 35 | For instance, a decoded AIS message will look something like this: 36 | 37 | `print ais.decode('15PIIv7P00D5i9HNn2Q3G?wB0t0I')` (nb the last portion of the message is the payload to be decoded. see the AIS spec for a fuller explanation.) 38 | 39 | ``` 40 | { 41 | u'slot_timeout':7, 42 | u'sync_state':1, 43 | u'received_stations':25, 44 | u'true_heading':511, 45 | u'sog':0.0, 46 | u'rot':-731.386474609375, 47 | u'nav_status':7, 48 | u'repeat_indicator':0, 49 | u'raim':False, 50 | u'id':1, 51 | u'spare':0, 52 | u'cog':86.0, 53 | u'timestamp':41, 54 | u'y':53.90443420410156, 55 | u'x':-166.5121307373047, 56 | u'position_accuracy':0, 57 | u'rot_over_range':True, 58 | u'mmsi':369515000, 59 | u'special_manoeuvre':0 60 | } 61 | ``` 62 | 63 | The MMSI field contains the ship's Maritime Mobile Service Identity number. The MMSI is a unique identifier for the ship's radio that can be used to communicate directly with the ship. MMSIs are regulated and managed internationally by the International Telecommunications Union in Geneva, Switzerland, just as radio call signs are regulated. 64 | 65 | 66 | ![Bill of Lading](https://s3.amazonaws.com/marcdacosta.com/storage/bill+of+lading.jpg) 67 | 68 | The `ship-utils.py` contains a function for looking up the MMSI with the ITU in order to find out the Vessel Name (you can also get the Vessel's satellite phone number if you're interested in giving them a call). If the ship is transporting containers or other commodities, the Vessel Name can be used to query the [Enigma Public API](https://public.enigma.com) to discover what the ship contains. 69 | 70 | The script relies upon data published in the Automated Manifest System by the US Customs and Border Protection. It contains structured copies of the bills of lading of everything that is imported into the United States. A bill of lading is a document that describes a shipment: who sent it, where its going, what it contains, &c. 71 | 72 | ### Origin: 73 | 74 | Ambient Shipping grew out of a residency of the [More&More Unlimited](http://www.moreandmore.world/) collective on [Governor's Island](https://www.google.com/maps/place/Governors+Island/@40.6885841,-74.0281299,15z/data=!3m1!4b1!4m5!3m4!1s0x89c25a7ada7d834f:0x78c2917911c7f535!8m2!3d40.6894501!4d-74.016792) in collaboration with [Surya Mattu](http://www.suryamattu.com/), [Sarah Rothberg](http://sarahrothberg.com/) and [Marina Zurkow](http://www.o-matic.com/) and it was made possible with support from the [Lower Manhattan Cultural Council](http://lmcc.net/). More&More Unlimited produces critical and artistic work engaging with the impact of maritime shipping on deep oceans and the logistics of global trade. 75 | -------------------------------------------------------------------------------- /ais-server.py: -------------------------------------------------------------------------------- 1 | #consider logging each raw NMEA message to a seperate pg table or local file 2 | 3 | import socket 4 | import ais.stream 5 | import logging 6 | import json 7 | import psycopg2 8 | import sys 9 | 10 | logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s') 11 | #logging.basicConfig(filename='ais-server.log', level=logging.DEBUG, format='%(asctime)s - %(levelname)s - %(message)s') 12 | 13 | try: 14 | from db_conf import * 15 | # db_conf file schema: 16 | # dbname = "" 17 | # user = "" 18 | # host = "" 19 | # pwd = "" 20 | except ImportError: 21 | logging.critical("could not load db conf file") 22 | sys.exit() 23 | 24 | try: 25 | conn = psycopg2.connect("dbname='" + dbname + "' user='" + user + "' host='" + host + "' password='" + pwd + "'") 26 | except: 27 | logging.critical("could not establish connection to db") 28 | 29 | cur = conn.cursor() 30 | 31 | UDP_IP_ADDRESS = "127.0.0.1" 32 | UDP_PORT_NO = 10110 33 | 34 | serverSock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) 35 | serverSock.bind((UDP_IP_ADDRESS, UDP_PORT_NO)) 36 | logging.debug("successful server intiation") 37 | 38 | 39 | def insertdata(data): 40 | pgpayload = ({"table":dbtable, "data":json.dumps(data)}) 41 | 42 | try: 43 | cur.execute("""INSERT into """ + pgpayload['table'] + """ VALUES (%(data)s)""",pgpayload) 44 | conn.commit() 45 | except psycopg2.Error as e: 46 | print e 47 | logging.warning("could not insert pgpayload into db") 48 | 49 | 50 | while True: 51 | try: 52 | data, addr = serverSock.recvfrom(1024) 53 | payload=data.split(",") 54 | 55 | messagecontainer = "" 56 | 57 | pad = int(payload[-1].split('*')[0][-1]) 58 | msglength = int(payload[1]) 59 | msgpart = int(payload[2]) 60 | 61 | if msglength == 1: 62 | rawmessage = payload[5] 63 | decodedmessage = ais.decode(rawmessage,pad) 64 | 65 | logging.info("SUCCESS: decoded message -> %s", str(decodedmessage)) 66 | print json.dumps(decodedmessage) 67 | insertdata(decodedmessage) 68 | 69 | messagecontainer = "" 70 | 71 | else: 72 | msgcomplete = 0 73 | messagecontainer += payload[5] 74 | 75 | while (msgcomplete == 0): 76 | data, addr = serverSock.recvfrom(1024) 77 | payload=data.split(",") 78 | rawmessage = payload[5] 79 | msglength = int(payload[1]) 80 | msgpart = int(payload[2]) 81 | 82 | messagecontainer += rawmessage 83 | 84 | logging.debug("incoming data -> %s", str(data)) 85 | logging.debug("message part -> %s", str(msgpart)) 86 | logging.debug("message length -> %s", str(msglength)) 87 | logging.debug("pad -> %s", str(pad)) 88 | logging.debug("raw message -> %s", str(rawmessage)) 89 | logging.debug("messagecontainer -> %s", str(messagecontainer)) 90 | 91 | 92 | if (msglength == msgpart): 93 | pad = int(payload[-1].split('*')[0][-1]) 94 | 95 | #remove escape from test udp 96 | messagecontainer = messagecontainer.replace("\\","") 97 | 98 | logging.debug("final pad -> %s", str(pad)) 99 | logging.debug("final messagecontainer -> %s", str(messagecontainer)) 100 | 101 | decodedmessage = ais.decode(messagecontainer,pad) 102 | 103 | logging.info("SUCCESS: decoded multipart message -> %s", str(decodedmessage)) 104 | print json.dumps(decodedmessage) 105 | insertdata(decodedmessage) 106 | 107 | messagecontainer = "" 108 | msgcomplete = 1 109 | 110 | except: 111 | logging.warning("failed to process message") 112 | 113 | 114 | # Note code above implemented a strip of "\" from incoming messages given need to add escape when socat-ing on terminal; should be removed in prod 115 | # Example 2 part message 116 | # echo "\!AIVDM,2,1,9,B,53nFBv01SJ")[0] 115 | print link 116 | vessel_name = link.split("&")[2].replace("+"," ") 117 | return vessel_name 118 | 119 | 120 | # These functions will enable you to go from an MMSI to a Vessel Name. 121 | # Query that vessel name to find its most recent port of call. 122 | # And then to discover the contents of the containers it unloaded at that port. 123 | vessel_name = get_vessel_details(mmsi) 124 | find_most_recent_arrival(vessel_name) 125 | print things_in_container(container_number, shipment_id) #should be applied for each in the results of `find_most_recent_arrival` 126 | 127 | 128 | 129 | --------------------------------------------------------------------------------