├── .gitignore ├── .gitmodules ├── .vscode ├── c_cpp_properties.json └── settings.json ├── README.adoc ├── doc └── images │ ├── AutoMerge.png │ ├── BioSLAM.png │ ├── MetaUnion.png │ ├── MrExplorer.png │ └── UnionLogo.png └── scripts ├── bin ├── enable_explorer.sh ├── enable_fakemerge.sh ├── enable_master_discovery.sh ├── enable_master_sync.sh ├── enable_simulator.sh └── env.sh ├── install_client.sh ├── install_server.sh ├── killmeta.sh ├── start_client.sh └── start_server.sh /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # C extensions 7 | *.so 8 | 9 | # Distribution / packaging 10 | .Python 11 | build/ 12 | develop-eggs/ 13 | dist/ 14 | downloads/ 15 | eggs/ 16 | .eggs/ 17 | lib/ 18 | lib64/ 19 | stack/ 20 | parts/ 21 | sdist/ 22 | var/ 23 | wheels/ 24 | pip-wheel-metadata/ 25 | share/python-wheels/ 26 | *.egg-info/ 27 | .installed.cfg 28 | *.egg 29 | MANIFEST 30 | 31 | # PyInstaller 32 | # Usually these files are written by a python script from a template 33 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 34 | *.manifest 35 | *.spec 36 | 37 | # Installer logs 38 | pip-log.txt 39 | pip-delete-this-directory.txt 40 | 41 | # Unit test / coverage reports 42 | htmlcov/ 43 | .tox/ 44 | .nox/ 45 | .coverage 46 | .coverage.* 47 | .cache 48 | nosetests.xml 49 | coverage.xml 50 | *.cover 51 | *.py,cover 52 | .hypothesis/ 53 | .pytest_cache/ 54 | 55 | # Translations 56 | *.mo 57 | *.pot 58 | 59 | # Django stuff: 60 | *.log 61 | local_settings.py 62 | db.sqlite3 63 | db.sqlite3-journal 64 | 65 | # Flask stuff: 66 | instance/ 67 | .webassets-cache 68 | 69 | # Scrapy stuff: 70 | .scrapy 71 | 72 | # Sphinx documentation 73 | docs/_build/ 74 | 75 | # PyBuilder 76 | target/ 77 | 78 | # Jupyter Notebook 79 | .ipynb_checkpoints 80 | 81 | # IPython 82 | profile_default/ 83 | ipython_config.py 84 | 85 | # pyenv 86 | .python-version 87 | 88 | # pipenv 89 | # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. 90 | # However, in case of collaboration, if having platform-specific dependencies or dependencies 91 | # having no cross-platform support, pipenv may install dependencies that don't work, or not 92 | # install all needed dependencies. 93 | #Pipfile.lock 94 | 95 | # PEP 582; used by e.g. github.com/David-OConnor/pyflow 96 | __pypackages__/ 97 | 98 | # Celery stuff 99 | celerybeat-schedule 100 | celerybeat.pid 101 | 102 | # SageMath parsed files 103 | *.sage.py 104 | 105 | # Environments 106 | .env 107 | .venv 108 | env/ 109 | venv/ 110 | ENV/ 111 | env.bak/ 112 | venv.bak/ 113 | 114 | # Spyder project settings 115 | .spyderproject 116 | .spyproject 117 | 118 | # Rope project settings 119 | .ropeproject 120 | 121 | # mkdocs documentation 122 | /site 123 | 124 | # mypy 125 | .mypy_cache/ 126 | .dmypy.json 127 | dmypy.json 128 | 129 | # Pyre type checker 130 | .pyre/ 131 | -------------------------------------------------------------------------------- /.gitmodules: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MetaSLAM/Ghostar/ea413de77bde8e4f8737ef40016f906482b33fb3/.gitmodules -------------------------------------------------------------------------------- /.vscode/c_cpp_properties.json: -------------------------------------------------------------------------------- 1 | { 2 | "configurations": [ 3 | { 4 | "name": "Linux", 5 | "includePath": [ 6 | "${workspaceFolder}/**", 7 | "/usr/include/eigen3" 8 | ], 9 | "defines": [], 10 | "compilerPath": "/usr/bin/gcc", 11 | "cStandard": "c11", 12 | "cppStandard": "gnu++14", 13 | "intelliSenseMode": "linux-gcc-x64" 14 | } 15 | ], 16 | "version": 4 17 | } -------------------------------------------------------------------------------- /.vscode/settings.json: -------------------------------------------------------------------------------- 1 | { 2 | "C_Cpp.errorSquiggles": "Disabled" 3 | } -------------------------------------------------------------------------------- /README.adoc: -------------------------------------------------------------------------------- 1 | :sectnums: 2 | :sectnumlevels: 1 3 | :toc: macro 4 | :toclevels: 2 5 | 6 | image::doc/images/UnionLogo.png["UnionLogo", width=800px] 7 | 8 | toc::[] 9 | 10 | == Overview 11 | In traditional Multi-Agent Exploration (MAE) methods, all agents start in the same place and start with a shared coordinate system; and once the relative transformation missed due to localization failures, the MAE system will be very fregial and easily crash. This framework provide a novel Autonomous Exploration paradigm, all the agents can start in the darkness, without knowing anything about *where they are*, *where each other are*, and *what the environment looks like*. As the MAE system explorer individually, *MetaUnion* helps to find associated overlaps among different agents and solve the above three challenges gradually; and most importantly, *MetaUnion* can incrementally update the feature learning ability for new observed areas. In a summary, our key contributions includes: 12 | 13 | * *Multi-Agent Localiztion without Initial Transformation;* 14 | * *Multi-Agent Hierarchical Exploration;* 15 | * *Incremental Lifelong Localization.* 16 | 17 | === System Structure 18 | 19 | image::doc/images/MetaUnion.png["MetaUnion", width=850px] 20 | 21 | MetaUnion mainly contains three key modules within the framework: *AutoMerge*, *MrExplorer*, and *BioSLAM*. In the begining, each agent is randomly assigned at a new envionrment, and has neither information about other agents' localization, nor place recognition ability for the new areas, and conduct single-agent exploration individually. 22 | When the trajectries of different agents have overlaps, *AutoMerge* module can automatically detect the data association between agents, and estimate the relative positions for them. Based on the relative transformation, *MrExplorer* can divide the agent into different groups based on their connections, and apply hierarichal exploration for individual agent. Parallel with the above procedure, *BioSLAM* constructs a dual-memory system to memorize the large-scale and long-duration place features. As the MAS system explorer to bigger area, their cooperation ability and place recognition ability are improved gradually. 23 | 24 | == Software 25 | 26 | To quickly install all the necessary softwares for MetaUnion, please following the instructions, 27 | [source,shell] 28 | ---- 29 | git clone --recurse-submodules -j8 https://github.com/MetaSLAM/MetaUnion.git 30 | # On Server 31 | sh scripts/install_server.sh 32 | ## On Client 33 | sh scripts/install_client.sh 34 | source ~/.bashrc 35 | ---- 36 | Please keep in mind, the scripts will automatically generate *PATH* into the **bashrc**, and you can use 37 | [source,bash] 38 | ---- 39 | METASLAM=/home/maxtom/codespace/MetaUnion 40 | source /home/maxtom/codespace/MetaUnion/stack/devel/setup.bash 41 | # on client 42 | alias enclient='sh /home/maxtom/codespace/MetaUnion/scripts/start_client.sh' 43 | # on server 44 | alias enserver='sh /home/maxtom/codespace/MetaUnion/scripts/start_server.sh' 45 | # general command 46 | alias cdw='cd /home/maxtom/codespace/MetaUnion' 47 | alias killmeta='sh /home/maxtom/codespace/MetaUnion/scripts/killall.sh' 48 | ---- 49 | In the following sections, we will investigate the details of each module. 50 | 51 | === MrExplorer 52 | 53 | image::doc/images/MrExplorer.png["System", width=800px] 54 | 55 | Since all agents begin from unknown darkness area within an given map, each agent are starting from a local exploration policy based on its observations. And based on the *AutoMerge* module, small group agents will find their relative transformations, then such agents are grouped into one set, and the single agent exploration policy are transfermed into multi-agent exploration policy. Gradually, all the agents are converted into one group. This hierarchical strategy help crowd agents quickly understand their coopeartion status with each others, and can balance the exploration efficiency and robustness without any initial position and relative transformation information. 56 | 57 | .API: [white blue-background]#*Input*# and [white red-background]#*Output*# 58 | [NOTE] 59 | ==== 60 | [options="interactive"] 61 | - [white blue-background]#*Odometry for each agent*# 62 | - [white blue-background]#*Grouping status from AutoMerge*# 63 | - [white red-background]#*Waypoints for each agent*# 64 | ==== 65 | 66 | === AutoMerge 67 | 68 | image::doc/images/AutoMerge.png["System", width=800px] 69 | 70 | AutoMerge can provide an automatic data-association for agents without any knowledge about initial position and each others. When agents upload the observations to the cloud, the agents can detect their associated overlaps (if exists), and send back the grouping results among different agent groups. 71 | 72 | .API: [white blue-background]#*Input*# and [white red-background]#*Output*# 73 | [NOTE] 74 | ==== 75 | [options="interactive"] 76 | - [white blue-background]#*Odometry from individual agents*# 77 | - [white blue-background]#*SubMaps from individual agents*# 78 | - [white red-background]#*Grouping Indexes*# 79 | - [white red-background]#*Relative Transformations*# 80 | ==== 81 | 82 | .TODO List 83 | [NOTE] 84 | ==== 85 | [options="interactive"] 86 | - [ ] [red yellow-background]#*Onine SubMap Generation*# 87 | - [ ] [red yellow-background]#*Onine Feature Evaluation*# 88 | - [ ] [red yellow-background]#*Onine Cache Checking*# 89 | - [ ] [red yellow-background]#*Onine Position Estimation*# 90 | ==== 91 | 92 | === BioSLAM 93 | 94 | image::doc/images/BioSLAM.png["System", width=800px] 95 | 96 | We present BioSLAM, a lifelong SLAM framework for learning various new appearances incrementally and maintaining accurate place recognition for previously visited areas.Unlike humans, artificial neural networks suffer from catastrophic forgetting and may forget the previously visited areas when trained with new arrivals. For humans, researchers discover that there exists a memory replay mechanism in the brain to keep the neuron active for previous events. Inspired by this discovery, BioSLAM designs a gated generative replay to control the robot's learning behavior based on the feedback rewards.Specifically, BioSLAM provides a novel dual-memory mechanism for maintenance: 1) a dynamic memory to efficiently learn new observations and 2) a static memory to balance new-old knowledge. When combined with a visual-/LiDAR- based SLAM system, the complete processing pipeline can help the agent incrementally update the place recognition ability, robust to the increasing complexity of long-term place recognition. 97 | 98 | .API: [white blue-background]#*Input*# and [white red-background]#*Output*# 99 | [NOTE] 100 | ==== 101 | [options="interactive"] 102 | - [white blue-background]#*Dual Memory Assistance*# 103 | - [white blue-background]#*Long-short Memorization*# 104 | - [white red-background]#*Regular Updated Models*# 105 | ==== 106 | 107 | .TODO List 108 | [NOTE] 109 | ==== 110 | [options="interactive"] 111 | - [ ] [red yellow-background]#*Online Feature Updating*# 112 | - [ ] [red yellow-background]#*Strategy Selection*# 113 | - [ ] [red yellow-background]#*Group Waypoints Estimation*# 114 | ==== 115 | 116 | == Multi-Agent Exploration 117 | 118 | === Run MrExplorer 119 | 120 | [source, bash] 121 | ---- 122 | cdw 123 | enserver #on server 124 | enclient #on client 125 | ---- 126 | 127 | .TODO List 128 | [NOTE] 129 | ==== 130 | [options="interactive"] 131 | - [ ] [red yellow-background]#*Change the name of the project*# 132 | - [ ] [red yellow-background]#*Integrate the configuration of the client into the config file*# 133 | - [ ] [red yellow-background]#*Debug the global planning*# 134 | - [ ] [red yellow-background]#*Integrate the multi-master into this*# 135 | ==== 136 | 137 | === FakeMerge 138 | 139 | [source, bash] 140 | ---- 141 | git clone https://github.com/MetaSLAM/fakemerge.git fakemerge 142 | cd fakemerge 143 | catkin_make 144 | source devel/setup.bash 145 | roslaunch fakemerge transformPose.launch 146 | ---- 147 | 148 | == Multi-Agent Localization 149 | 150 | [source,yaml] 151 | ---- 152 | DATA: 153 | OFFLINE_LENGTH: 100 # Set for desire testing length 154 | ---- 155 | Use `OFFLINE_LENGTH` to set the length for each agent. 156 | 157 | [source,bash] 158 | ---- 159 | python src/offline_merging.py 160 | ---- 161 | 162 | In the current `global_rough_align` step, we will use spectral clustering method to divide agents into different groups based on their connections. 163 | 164 | === Visualization 165 | 166 | Outside the docker, subscibe `/global_map` with `world` frame with `rviz`. Different un-merged maps will be visualized along the z-axis (`index*30`). 167 | 168 | == Lifelong Localization 169 | 170 | See the demo tutorial in the link:tests/demo.ipynb[jupyter] and link:tests/test.py[python] version. 171 | 172 | === Configure Datasets 173 | 174 | Download Pittsburgh datasets for online map merging. 175 | https://drive.google.com/drive/folders/19AK8jc6yZpKN6Ub_ILGJaceZixpbwKcV?usp=sharing 176 | 177 | Download pre-trained models and set path 178 | [source,bash] 179 | ---- 180 | cd data && sh download.sh 181 | echo "export ROS_IP='172.17.0.1' " >> ~/.bashrc 182 | echo "export MASTER_IP='172.17.0.2' " >> ~/.bashrc 183 | echo "export ROS_MASTER_URI=http://$MASTER_IP:11311/ " >> ~/.bashrc 184 | echo "export BAG_PATH='$PATH_TO_PITT_ROS_BAGS' " >> ~/.bashrc 185 | source ~/.bashrc 186 | ---- 187 | And the following difference matrix in the `data/results/`. Then set `ROS_IP` to enable communication between docker and host computer. 188 | Finally, export dataset path, and replace `PATH_TO_PITT_ROS_BAGS` to the Pittusbrugh rosbags. 189 | 190 | 191 | == TODO List 192 | -------------------------------------------------------------------------------- /doc/images/AutoMerge.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MetaSLAM/Ghostar/ea413de77bde8e4f8737ef40016f906482b33fb3/doc/images/AutoMerge.png -------------------------------------------------------------------------------- /doc/images/BioSLAM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MetaSLAM/Ghostar/ea413de77bde8e4f8737ef40016f906482b33fb3/doc/images/BioSLAM.png -------------------------------------------------------------------------------- /doc/images/MetaUnion.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MetaSLAM/Ghostar/ea413de77bde8e4f8737ef40016f906482b33fb3/doc/images/MetaUnion.png -------------------------------------------------------------------------------- /doc/images/MrExplorer.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MetaSLAM/Ghostar/ea413de77bde8e4f8737ef40016f906482b33fb3/doc/images/MrExplorer.png -------------------------------------------------------------------------------- /doc/images/UnionLogo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MetaSLAM/Ghostar/ea413de77bde8e4f8737ef40016f906482b33fb3/doc/images/UnionLogo.png -------------------------------------------------------------------------------- /scripts/bin/enable_explorer.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | source $METASLAM/scripts/bin/env.sh 3 | roslaunch tare_planner explore_campus.launch -------------------------------------------------------------------------------- /scripts/bin/enable_fakemerge.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | source $METASLAM/scripts/bin/env.sh 3 | roslaunch fakemerge transformPose.launch -------------------------------------------------------------------------------- /scripts/bin/enable_master_discovery.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | source $METASLAM/scripts/bin/env.sh 3 | roslaunch fkie_master_discovery master_discovery.launch -------------------------------------------------------------------------------- /scripts/bin/enable_master_sync.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | source $METASLAM/scripts/bin/env.sh 3 | roslaunch fkie_multimaster_sync master_sync.launch -------------------------------------------------------------------------------- /scripts/bin/enable_simulator.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | source $METASLAM/scripts/bin/env.sh 3 | roslaunch vehicle_simulator system_campus.launch -------------------------------------------------------------------------------- /scripts/bin/env.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | source /opt/ros/melodic/setup.bash 3 | source $METASLAM/stack/devel/setup.bash 4 | export ROS_IP='192.168.8.182' 5 | export ROS_MASTER_URI=http://192.168.8.182:11311 -------------------------------------------------------------------------------- /scripts/install_client.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | METASLAM=$(pwd) 4 | 5 | # Install Necessay packages 6 | sudo apt-get install libusb-dev python-pip screen tmux python-rosdep 7 | sudo pip install catkin-tools 8 | 9 | # Install Simulation 10 | git clone https://github.com/MetaSLAM/Multi-Agent-Simulation.git $METASLAM/stack/src/simulation 11 | git clone https://github.com/fkie/multimaster_fkie.git $METASLAM/stack/src/multimaster 12 | git clone https://github.com/MetaSLAM/MrExplorer.git $METASLAM/stack/src/mrexplorer 13 | git clone https://github.com/MetaSLAM/fakemerge.git $METASLAM/stack/src/fakemerge 14 | 15 | MM_PACK=$METASLAM/stack/multimaster/ 16 | rosdep install -i --as-root pip:false --reinstall --from-paths $MM_PACK 17 | 18 | cd $METASLAM/stack/src/simulation/src/vehicle_simulator/mesh/ 19 | sh download_environments.sh 20 | 21 | cd $METASLAM/stack 22 | catkin build -j 8 23 | 24 | echo "source $METASLAM/stack/devel/setup.bash" >> ~/.bashrc 25 | 26 | # Need to modify based on multi-agent 27 | echo "export ROS_IP='172.26.169.231'" >> ~/.bashrc 28 | echo "export ROS_MASTER_URI=http://172.26.169.231:11311" >> ~/.bashrc 29 | 30 | echo "METASLAM=$METASLAM" >> ~/.bashrc 31 | echo "alias cdw='cd $METASLAM'" >> ~/.bashrc 32 | echo "alias enclient='sh $METASLAM/scripts/start_client.sh'" >> ~/.bashrc 33 | echo "alias killmeta='pkill screen'" >> ~/.bashrc -------------------------------------------------------------------------------- /scripts/install_server.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | METASLAM=$(pwd) 4 | 5 | # Install Necessay packages 6 | sudo apt-get install libusb-dev python-pip screen tmux 7 | sudo pip install catkin-tools 8 | 9 | # Get All 10 | git clone https://github.com/MetaSLAM/Multi-Agent-Simulation.git $METASLAM/stack/src/simulation 11 | git clone https://github.com/fkie/multimaster_fkie.git $METASLAM/stack/src/multimaster 12 | git clone https://github.com/MetaSLAM/MrExplorer.git $METASLAM/stack/src/mrexplorer 13 | git clone https://github.com/MetaSLAM/fakemerge.git $METASLAM/stack/src/fakemerge 14 | git clone https://github.com/MetaSLAM/AutoMerge.git $METASLAM/stack/src/automerge 15 | 16 | # Install python env for multimaster 17 | MM_PACK=$METASLAM/stack/multimaster/ 18 | rosdep install -i --as-root pip:false --reinstall --from-paths $MM_PACK 19 | 20 | # Download sim environments 21 | cd $METASLAM/stack/src/simulation/src/vehicle_simulator/mesh/ 22 | sh download_environments.sh 23 | 24 | # Install AutoMerge 25 | AUTOMERGE_PATH=$METASLAM/stack/src/automerge 26 | cd $AUTOMERGE_PATH 27 | # pip install -r requirements 28 | 29 | # Install BioSLAM 30 | 31 | # Compile ROS Env 32 | cd $METASLAM/stack 33 | catkin build -j 8 34 | 35 | #Setup Environment Parameters 36 | echo "source $METASLAM/stack/devel/setup.bash" >> ~/.bashrc 37 | echo "METASLAM=$METASLAM" >> ~/.bashrc 38 | echo "alias cdw='cd $METASLAM'" >> ~/.bashrc 39 | echo "alias enclient='sh $METASLAM/scripts/start_client.sh'" >> ~/.bashrc 40 | echo "alias killmeta='pkill screen'" >> ~/.bashrc -------------------------------------------------------------------------------- /scripts/killmeta.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | pkill screen -------------------------------------------------------------------------------- /scripts/start_client.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | export METASLAM=$(pwd) 3 | 4 | screen -Sdm start_simulator sh $METASLAM/scripts/bin/enable_simulator.sh 5 | echo "Enable Simulator" 6 | sleep 10 7 | 8 | screen -Sdm start_mr_explorer sh $METASLAM/scripts/bin/enable_explorer.sh 9 | echo "Enable Mr Explorer" 10 | sleep 1 11 | -------------------------------------------------------------------------------- /scripts/start_server.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | screen -Sdm start_simulator sh $METASLAM/scripts/bin/enable_simulator.sh 4 | echo "Enable Simulator" 5 | sleep 1 6 | 7 | screen -Sdm start_ms_discover sh $METASLAM/scripts/bin/enable_master_discovery.sh 8 | echo "Enable Master Discovery" 9 | sleep 1 10 | 11 | screen -Sdm start_ms_sync sh $METASLAM/scripts/bin/enable_master_sync.sh 12 | echo "Enable Master Sync" 13 | sleep 1 14 | 15 | 16 | screen -Sdm start_mr_explorer sh $METASLAM/scripts/bin/enable_explorer.sh 17 | echo "Enable Mr Explorer" 18 | sleep 1 19 | 20 | screen -Sdm start_fakemerge sh $METASLAM/scripts/bin/enable_fakemerge.sh 21 | echo "Enable Fake Merging" 22 | sleep 1 23 | --------------------------------------------------------------------------------