├── .gitignore
├── README.md
├── aws_instance.py
├── spacecore.cfg.default
└── spacecore.sh


/.gitignore:
--------------------------------------------------------------------------------
1 | spacecore.cfg
2 | 


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
  1 | # SpaceCore
  2 | 
  3 | Spacecore is a script allowing to start, stop and execute script on a
  4 | aws instance of amazon.
  5 | 
  6 | # Example
  7 | ```
  8 | $ spacecore start
  9 | Starting Server...OK!
 10 | [-] Current server state: running
 11 | Allocating IP Address...OK! (ip: 42.42.42.42)
 12 | Associating server to ip...OK!
 13 | Done.
 14 | 
 15 | $ spacecore run keras-mnist-gpu.py
 16 | Sending file keras-mnist-gpu.py to server (42.42.42.42)...
 17 | Warning: Permanently added '42.42.42.42' (ECDSA) to the list of known hosts.
 18 | keras-mnist-gpu.py                                                     100% 2073    65.3KB/s   00:00    
 19 | Using TensorFlow backend.
 20 | I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcublas.so locally
 21 | I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcudnn.so locally
 22 | I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcufft.so locally
 23 | I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcuda.so.1 locally
 24 | I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcurand.so locally
 25 | ...
 26 | 
 27 | $ spacecore stop
 28 | Stopping Server...OK!
 29 | Disassociating server from ip...OK!
 30 | Releasing IP Address...OK!
 31 | Done.
 32 | ```
 33 | 
 34 | ## Dependencies
 35 | You will need to install the aws command line (http://docs.aws.amazon.com/cli/latest/userguide/installing.html) and boto3 python lib:
 36 | ```
 37 | sudo pip3 install boto3
 38 | ```
 39 | 
 40 | ## Installation
 41 | Copy the folder spacecore in /opt and add the following line to your .bashrc
 42 | ```
 43 | alias spacecore=/opt/spacecore/spacecore.sh
 44 | ```
 45 | Also, don't forget to save your aws credentials for the api with
 46 | ```
 47 | aws configure
 48 | ```
 49 | Now, you will have to create an aws instance (g2.2xlarge for instance), and there
 50 | is no need to associate an IP address, the script will create an elastic IP and
 51 | destroy it when you shutdown the instance. Also, write down the Instance ID.
 52 | 
 53 | The final step is to configure the spacecore.cfg file. First, you will need to
 54 | copy or rename spacecore.cfg.default into spacecore.cfg. There are several
 55 | variables that you can configure:
 56 | * _sv_user_ User name of the aws instance
 57 | * _sv_path_ Path to the workspace of the script
 58 | * _sv_interpreter_ Interpreter to use on the server (default: python3)
 59 | * _cl_key_ Link to the key file (extension .pem)
 60 | * _cl_id_ Instance ID that you just wrote down
 61 | 
 62 | It is done, have fun!
 63 | 
 64 | ## Usage
 65 | ### Starting the instance
 66 | ```
 67 | spacecore start
 68 | ```
 69 | Starts the server, create the elastic IP and associate it.
 70 | 
 71 | Takes some time, no worries ;)
 72 | 
 73 | ### Stopping the instance
 74 | ```
 75 | spacecore stop
 76 | ```
 77 | Stops the server and deletes the elastic IP so you don't have to pay for it.
 78 | 
 79 | ### Sending a file
 80 | ```
 81 | spacecore send <file>
 82 | ```
 83 | Uploads a file in the workspace.
 84 | 
 85 | ### Downloading a file
 86 | ```
 87 | spacecore get <file>
 88 | ```
 89 | Download a file from the workspace.
 90 | 
 91 | ### Running a script
 92 | ```
 93 | spacecore run <script.py>
 94 | ```
 95 | Uploads a python script and executes it with the command:
 96 | ```
 97 | sudo -i python3 <script.py>
 98 | ```
 99 | If you are unsatisfied with this command you can edit it in the spacecore.sh file.
100 | The sudo part of the command allows the script to use cuda on the instance.
101 | 
102 | ### Getting the IP address
103 | ```
104 | spacecore ip
105 | ```
106 | Returns the IP address only. If you program anything related to the instance,
107 | you should use this command as the IP is going to change at every restart.
108 | 
109 | For example (ping):
110 | ```
111 | ping $(spacecore ip)
112 | ```
113 | 
114 | ### Getting the status of the instance
115 | ```
116 | spacecore status
117 | ```
118 | Returns the current status of the instance.
119 | The output is only one word, so this can be easily used in other scripts.
120 | 
121 | ### Starting a SSH session
122 | ```
123 | spacecore ssh
124 | ```
125 | Starts a SSH session directly in the terminal.
126 | 


--------------------------------------------------------------------------------
/aws_instance.py:
--------------------------------------------------------------------------------
  1 | #!/usr/bin/env python3
  2 | # Author: Nicolas Pielawski
  3 | # Creation date: May 13 2016
  4 | import boto3
  5 | import json
  6 | import sys
  7 | import time
  8 | 
  9 | client = boto3.client('ec2')
 10 | ec2 = boto3.resource('ec2')
 11 | 
 12 | 
 13 | def allocate_elastic_ip():
 14 |     result = client.allocate_address()
 15 |     return (result["PublicIp"], result["AllocationId"])
 16 | 
 17 | 
 18 | def associate_elastic_ip(serverid, ip):
 19 |     return client.associate_address(InstanceId=serverid, PublicIp=ip)
 20 | 
 21 | 
 22 | def disassociate_elastic_ip(ip):
 23 |     return client.disassociate_address(PublicIp=ip)
 24 | 
 25 | 
 26 | def release_elastic_ip(ip):
 27 |     result = client.describe_addresses(PublicIps=[ip])
 28 |     alloc_id = result["Addresses"][0]["AllocationId"]
 29 |     return client.release_address(AllocationId=alloc_id)
 30 | 
 31 | 
 32 | def server_status(server_id):
 33 |     result = client.describe_instances(InstanceIds=[server_id])
 34 |     if len(result["Reservations"]) == 0:
 35 |         return "N/A"
 36 |     return result["Reservations"][0]["Instances"][0]["State"]["Name"]
 37 | 
 38 | 
 39 | def server_ip(server_id):
 40 |     result = client.describe_instances(InstanceIds=[server_id])
 41 |     if len(result["Reservations"]) == 0:
 42 |         return "N/A"
 43 |     if "PublicIpAddress" in result["Reservations"][0]["Instances"][0]:
 44 |         return result["Reservations"][0]["Instances"][0]["PublicIpAddress"]
 45 |     else:
 46 |         return "N/A"
 47 | 
 48 | 
 49 | def wait_for_state(waited_state, time_to_wait=1):
 50 |     spin = 0
 51 |     spins = "-\|/"
 52 |     state = server_status(server_id)
 53 |     print("[{}] Current server state: {}".format(spins[spin], state),
 54 |           end='\r')
 55 |     spin += 1
 56 |     while state != waited_state:
 57 |         time.sleep(time_to_wait)
 58 |         state = server_status(server_id)
 59 |         print("[{}] Current server state: {}".format(spins[spin], state),
 60 |               end='\r')
 61 |         spin = (spin + 1) % len(spins)
 62 |     print()
 63 | 
 64 | 
 65 | def server_start(server_id):
 66 |     if server_status(server_id) == "stopping":
 67 |         print("The server is still stopping! Waiting...")
 68 |         wait_for_state("stopped")
 69 | 
 70 |     print("Starting Server...", end="")
 71 |     client.start_instances(InstanceIds=[server_id])
 72 |     print("OK!")
 73 | 
 74 |     wait_for_state("running")
 75 | 
 76 |     print("Allocating IP Address...", end="")
 77 |     (ip, alloc_id) = allocate_elastic_ip()
 78 |     print("OK! (ip: " + ip + ")")
 79 | 
 80 |     print("Associating server to ip...", end="")
 81 |     associate_elastic_ip(server_id, ip)
 82 |     print("OK!")
 83 | 
 84 |     return (ip, alloc_id)
 85 | 
 86 | 
 87 | def server_stop(server_id):
 88 |     ip = server_ip(server_id)
 89 | 
 90 |     print("Stopping Server...", end="")
 91 |     client.stop_instances(InstanceIds=[server_id])
 92 |     print("OK!")
 93 | 
 94 |     if ip == "N/A":
 95 |         print("No IP address associated!")
 96 |         return
 97 | 
 98 |     print("Disassociating server from ip...", end="")
 99 |     disassociate_elastic_ip(ip)
100 |     print("OK!")
101 | 
102 |     print("Releasing IP Address...", end="")
103 |     release_elastic_ip(ip)
104 |     print("OK!")
105 | 
106 | 
107 | if __name__ == '__main__':
108 |     if len(sys.argv) < 3:
109 |         print("You have to specify the client-id and"
110 |               "action to perform (start/stop)")
111 |     else:
112 |         action = sys.argv[1]
113 |         server_id = sys.argv[2]
114 |         if action == "start":
115 |             server_start(server_id)
116 |         elif action == "stop":
117 |             server_stop(server_id)
118 |         elif action == "status":
119 |             print(server_status(server_id))
120 |         elif action == "ip":
121 |             print(server_ip(server_id))
122 |         else:
123 |             print("Unknown action...")
124 | 


--------------------------------------------------------------------------------
/spacecore.cfg.default:
--------------------------------------------------------------------------------
 1 | # SERVER SIDE
 2 | # Username on the server
 3 | sv_user="ubuntu"
 4 | # Workspace path on the server
 5 | sv_path="/home/$sv_user"
 6 | # Interpreter to use
 7 | sv_interpreter="python3"
 8 | 
 9 | # CLIENT SIDE
10 | # Key to connect to the server
11 | cl_key="/home/user/.ssh/yourkey.pem"
12 | # Instance ID of the server
13 | cl_id="i-00000000"
14 | 


--------------------------------------------------------------------------------
/spacecore.sh:
--------------------------------------------------------------------------------
  1 | #!/bin/bash
  2 | # Author: Nicolas Pielawski
  3 | # Creation date: May 13 2016
  4 | source /opt/spacecore/spacecore.cfg
  5 | 
  6 | # Is there an argument ?
  7 | if [ $# -eq 0 -o "$1" == "help" ]
  8 | then
  9 |     echo "What would you like to do, sir ?"
 10 |     echo "* start: start the instance"
 11 |     echo "* stop: stop the instance"
 12 |     echo "* run: send and run a python file"
 13 |     echo "* send: send a file to the server"
 14 |     echo "* get: download a file from the server"
 15 |     echo "* status: get the status of the server"
 16 |     echo "* ip: get the IP of the server"
 17 |     echo "* ssh: start a SSH client"
 18 |     echo "* help: print this help"
 19 |     exit
 20 | fi
 21 | 
 22 | case "$1" in
 23 |     "start")
 24 |         # Starting server
 25 |         /opt/spacecore/aws_instance.py start $cl_id
 26 |         echo "Done."
 27 |         ;;
 28 |     "stop")
 29 |         # Stopping server
 30 |         /opt/spacecore/aws_instance.py stop $cl_id
 31 |         echo "Done."
 32 |         ;;
 33 |     "run")
 34 |         if [ -z $2 ]
 35 |         then
 36 |             echo "You must give the script to compute"
 37 |             exit
 38 |         elif [ ! -e $2 ]
 39 |         then
 40 |             echo "File $2 doesn't exist (check your rights, too)"
 41 |             exit
 42 |         fi
 43 |         if [ $(/opt/spacecore/aws_instance.py status $cl_id) != "running" ]
 44 |         then
 45 |             echo "Server is not running yet!"
 46 |             exit
 47 |         fi
 48 |         sv_address=$(/opt/spacecore/aws_instance.py ip $cl_id)
 49 |         echo "Sending file $2 to server ($sv_address)..."
 50 |         scp -o "StrictHostKeyChecking no" -i $cl_key $2 $sv_user@$sv_address:$sv_path/$(basename $2)
 51 |         svcmd="sudo -i bash -c "'"'"cd $sv_path; $sv_interpreter $sv_path/$(basename $2)"'"'
 52 |         ssh -o "StrictHostKeyChecking no" -i $cl_key -t $sv_user@$sv_address "$svcmd"
 53 |         ;;
 54 |     "send")
 55 |         if [ -z $2 ]
 56 |         then
 57 |             echo "You must give the file to send"
 58 |             exit
 59 |         elif [ ! -e $2 ]
 60 |         then
 61 |             echo "File $2 doesn't exist (check your rights, too)"
 62 |             exit
 63 |         fi
 64 |         if [ $(/opt/spacecore/aws_instance.py status $cl_id) != "running" ]
 65 |         then
 66 |             echo "Server is not running yet!"
 67 |             exit
 68 |         fi
 69 |         sv_address=$(/opt/spacecore/aws_instance.py ip $cl_id)
 70 |         echo "Sending file $2 to server ($sv_address)..."
 71 |         scp -i $cl_key $2 $sv_user@$sv_address:$sv_path/$(basename $2)
 72 |         ;;
 73 |     "get")
 74 |         if [ -z $2 ]
 75 |         then
 76 |             echo "You must name the file to download"
 77 |             exit
 78 |         fi
 79 |         if [ $(/opt/spacecore/aws_instance.py status $cl_id) != "running" ]
 80 |         then
 81 |             echo "Server is not running yet!"
 82 |             exit
 83 |         fi
 84 |         sv_address=$(/opt/spacecore/aws_instance.py ip $cl_id)
 85 |         echo "Downloading file $2 from server ($sv_address)..."
 86 |         scp -i $cl_key $sv_user@$sv_address:$sv_path/$2 .
 87 |         ;;
 88 |     "status")
 89 |         /opt/spacecore/aws_instance.py status $cl_id
 90 |         ;;
 91 |     "ip")
 92 |         /opt/spacecore/aws_instance.py ip $cl_id
 93 |         ;;
 94 |     "ssh")
 95 |         if [ $(/opt/spacecore/aws_instance.py status $cl_id) != "running" ]
 96 |         then
 97 |             echo "Server is not running yet!"
 98 |             exit
 99 |         fi
100 |         sv_address=$(/opt/spacecore/aws_instance.py ip $cl_id)
101 |         echo "Trying to connect to $sv_address..."
102 |         ssh -o "StrictHostKeyChecking no" -i $cl_key -t $sv_user@$sv_address
103 |         ;;
104 |     "help")
105 |         ;;
106 |     *)
107 |         echo "Invalid argument"
108 |         ;;
109 | esac
110 | 


--------------------------------------------------------------------------------