├── .mypy.ini ├── requirements.txt ├── .gitignore ├── docs ├── imgs │ ├── overview.drawio.png │ └── overview.drawio ├── source │ ├── modules │ │ ├── battery.rst │ │ ├── simulation.rst │ │ └── environment.rst │ ├── conf.py │ └── index.rst ├── Makefile └── make.bat ├── .readthedocs.yml ├── .ruff.toml ├── .github └── workflows │ └── ci.yml ├── CONTRIBUTING.md ├── LICENCE ├── tests ├── test_battery.py ├── test_building_simulation.py └── test_environment.py ├── setup.py ├── example_solutions ├── helper.py ├── observation_wrapper.py ├── deep_reinforcement_learning │ ├── train.py │ └── evaluate.py ├── optimal_control_problem.py └── model_predictive_control.py └── README.md /.mypy.ini: -------------------------------------------------------------------------------- 1 | [mypy] 2 | disallow_untyped_defs = True -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | pandas 2 | numpy 3 | gymnasium 4 | sphinx 5 | pytest 6 | stable-baselines3 7 | pyomo -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | .idea 2 | __pycache__ 3 | **.egg-info 4 | .ipynb_checkpoints 5 | docs/build 6 | build/ 7 | dist/ -------------------------------------------------------------------------------- /docs/imgs/overview.drawio.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tobirohrer/building-energy-storage-simulation/HEAD/docs/imgs/overview.drawio.png -------------------------------------------------------------------------------- /docs/source/modules/battery.rst: -------------------------------------------------------------------------------- 1 | .. _battery: 2 | 3 | Battery 4 | ======= 5 | 6 | .. autoclass:: building_energy_storage_simulation.Battery 7 | :members: -------------------------------------------------------------------------------- /docs/source/modules/simulation.rst: -------------------------------------------------------------------------------- 1 | .. _simulation: 2 | 3 | BuildingSimulation 4 | ================== 5 | 6 | .. autoclass:: building_energy_storage_simulation.BuildingSimulation 7 | :members: -------------------------------------------------------------------------------- /docs/source/modules/environment.rst: -------------------------------------------------------------------------------- 1 | .. _environment: 2 | 3 | Environment 4 | =========== 5 | 6 | .. autoclass:: building_energy_storage_simulation.Environment 7 | :members: 8 | :exclude-members: render 9 | -------------------------------------------------------------------------------- /.readthedocs.yml: -------------------------------------------------------------------------------- 1 | # Read the Docs configuration file 2 | # See https://docs.readthedocs.io/en/stable/config-file/v2.html for details 3 | 4 | # Required 5 | version: 2 6 | 7 | # Build documentation in the docs/ directory with Sphinx 8 | sphinx: 9 | configuration: docs/source/conf.py 10 | 11 | python: 12 | install: 13 | - method: pip 14 | path: . 15 | -------------------------------------------------------------------------------- /.ruff.toml: -------------------------------------------------------------------------------- 1 | # Same as Black. 2 | line-length = 127 3 | # Assume Python 3.8 4 | target-version = "py38" 5 | # See https://beta.ruff.rs/docs/rules/ 6 | select = ["E", "F", "B", "UP", "C90", "RUF", "I", "N", "B", "SIM"] 7 | # B028: Ignore explicit stacklevel` 8 | # RUF013: Too many false positives (implicit optional) 9 | ignore = ["B028", "RUF013"] 10 | 11 | [extend-per-file-ignores] 12 | "__init__.py" = ["F401", "F403"] 13 | -------------------------------------------------------------------------------- /.github/workflows/ci.yml: -------------------------------------------------------------------------------- 1 | name: CI 2 | 3 | on: 4 | push: 5 | branches: [ "master" ] 6 | pull_request: 7 | branches: [ "master" ] 8 | 9 | jobs: 10 | checkup: 11 | runs-on: ubuntu-latest 12 | 13 | steps: 14 | - uses: actions/checkout@v3 15 | 16 | - name: install dependencies 17 | run: pip install .[dev] 18 | 19 | - name: Run Tests with pytest 20 | run: pytest ./tests 21 | 22 | - name: Type checking with mypy 23 | run: mypy building_energy_storage_simulation 24 | 25 | - name: Code style checking with ruff 26 | run: ruff check . -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing to this Project 2 | 3 | As i just started with this project, I am very happy for any kind of 4 | contribution ! In case you want to contribute, or you have any 5 | questions, contact me via [discord](https://discord.com/users/tobirohrer#8654) 6 | 7 | If you just want to get started, please send a 8 | Pull Request to 9 | [https://github.com/tobirohrer/building-energy-storage-simulation](https://github.com/tobirohrer/building-energy-storage-simulation). 10 | 11 | Before you start developing, please create an 12 | [issue](https://github.com/tobirohrer/building-energy-storage-simulation/issues) 13 | first, or contact me via 14 | [discord](https://discord.com/users/tobirohrer#8654). 15 | -------------------------------------------------------------------------------- /docs/Makefile: -------------------------------------------------------------------------------- 1 | # Minimal makefile for Sphinx documentation 2 | # 3 | 4 | # You can set these variables from the command line, and also 5 | # from the environment for the first two. 6 | SPHINXOPTS ?= 7 | SPHINXBUILD ?= sphinx-build 8 | SOURCEDIR = source 9 | BUILDDIR = build 10 | 11 | # Put it first so that "make" without argument is like "make help". 12 | help: 13 | @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) 14 | 15 | .PHONY: help Makefile 16 | 17 | # Catch-all target: route all unknown targets to Sphinx using the new 18 | # "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). 19 | %: Makefile 20 | @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) 21 | -------------------------------------------------------------------------------- /docs/make.bat: -------------------------------------------------------------------------------- 1 | @ECHO OFF 2 | 3 | pushd %~dp0 4 | 5 | REM Command file for Sphinx documentation 6 | 7 | if "%SPHINXBUILD%" == "" ( 8 | set SPHINXBUILD=sphinx-build 9 | ) 10 | set SOURCEDIR=source 11 | set BUILDDIR=build 12 | 13 | %SPHINXBUILD% >NUL 2>NUL 14 | if errorlevel 9009 ( 15 | echo. 16 | echo.The 'sphinx-build' command was not found. Make sure you have Sphinx 17 | echo.installed, then set the SPHINXBUILD environment variable to point 18 | echo.to the full path of the 'sphinx-build' executable. Alternatively you 19 | echo.may add the Sphinx directory to PATH. 20 | echo. 21 | echo.If you don't have Sphinx installed, grab it from 22 | echo.https://www.sphinx-doc.org/ 23 | exit /b 1 24 | ) 25 | 26 | if "%1" == "" goto help 27 | 28 | %SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% 29 | goto end 30 | 31 | :help 32 | %SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% 33 | 34 | :end 35 | popd 36 | -------------------------------------------------------------------------------- /docs/source/conf.py: -------------------------------------------------------------------------------- 1 | # Configuration file for the Sphinx documentation builder. 2 | # 3 | # For the full list of built-in configuration values, see the documentation: 4 | # https://www.sphinx-doc.org/en/master/usage/configuration.html 5 | 6 | # -- Project information ----------------------------------------------------- 7 | # https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information 8 | 9 | project = 'building-energy-storage-simulation' 10 | copyright = '2022, tobirohrer' 11 | author = 'tobirohrer' 12 | release = '0.1' 13 | 14 | # -- General configuration --------------------------------------------------- 15 | # https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration 16 | 17 | extensions = [ 18 | 'sphinx.ext.autodoc', 19 | 'sphinx.ext.autosummary', 20 | ] 21 | 22 | templates_path = ['_templates'] 23 | exclude_patterns = [] 24 | 25 | 26 | # -- Options for HTML output ------------------------------------------------- 27 | # https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output 28 | 29 | html_theme = 'alabaster' 30 | html_static_path = ['_static'] 31 | -------------------------------------------------------------------------------- /LICENCE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2020 Jose Ramon Vazquez-Canteli, Intelligent Environments Laboratory 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. -------------------------------------------------------------------------------- /tests/test_battery.py: -------------------------------------------------------------------------------- 1 | from building_energy_storage_simulation import Battery 2 | 3 | 4 | def test_battery_charge_electricity_usage(): 5 | battery = Battery(capacity=100) # in kWh 6 | electricity_used = battery.use(10) 7 | electricity_used += battery.use(10) 8 | assert electricity_used == 20.0 9 | 10 | 11 | def test_battery_charge_state(): 12 | battery = Battery(capacity=100) 13 | electricity_used = battery.use(10) 14 | electricity_used += battery.use(10) 15 | assert battery.state_of_charge == 20.0 16 | 17 | 18 | def test_battery_use(): 19 | battery = Battery(capacity=100, initial_state_of_charge=100) 20 | electricity_used = battery.use(-10) 21 | assert electricity_used == -10.0 22 | 23 | 24 | def test_battery_empty_state_of_charge(): 25 | battery = Battery(capacity=100, initial_state_of_charge=100, max_battery_charge_per_timestep=10) 26 | battery.use(-10) 27 | battery.use(-10) 28 | assert battery.state_of_charge == 80 29 | 30 | 31 | def test_battery_max_charge(): 32 | battery = Battery(capacity=100, initial_state_of_charge=100, max_battery_charge_per_timestep=10) 33 | battery.use(-20) 34 | battery.use(-50) 35 | assert battery.state_of_charge == 80 36 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | from distutils.core import setup 2 | from pathlib import Path 3 | 4 | this_directory = Path(__file__).parent 5 | long_description = (this_directory / "README.md").read_text() 6 | 7 | setup(name='building_energy_storage_simulation', 8 | version='0.9.3', 9 | description='A simulation of a building to optimize energy storage utilization.', 10 | long_description=long_description, 11 | long_description_content_type='text/markdown', 12 | author='Tobias Rohrer', 13 | author_email='tobias.rohrer@outlook.com', 14 | url="https://github.com/tobirohrer/building-energy-storage-simulation", 15 | packages=['building_energy_storage_simulation'], 16 | package_dir={'building_energy_storage_simulation': 'building_energy_storage_simulation'}, 17 | # Required to include profiles which are stored as .csv files 18 | package_data={ 19 | "building_energy_storage_simulation": [ 20 | "data/preprocessed/*.csv", 21 | ] 22 | }, 23 | include_package_data=True, 24 | install_requires=[ 25 | "gymnasium", 26 | "pandas", 27 | "numpy" 28 | ], 29 | extras_require={ 30 | "dev": [ 31 | "sphinx", 32 | "pytest", 33 | "mypy", 34 | "pandas-stubs", 35 | "types-setuptools", 36 | "ruff" 37 | ] 38 | } 39 | ) 40 | -------------------------------------------------------------------------------- /example_solutions/helper.py: -------------------------------------------------------------------------------- 1 | from pathlib import Path 2 | from typing import Tuple 3 | 4 | import numpy as np 5 | import pandas as pd 6 | from matplotlib import pyplot as plt 7 | 8 | # Start and end Index of data used for testing 9 | TEST_INDEX_START = 4380 10 | TEST_INDEX_END = 8500 11 | 12 | BATTERY_CAPACITY = 400 13 | BATTERY_POWER = 100 14 | 15 | 16 | def read_data() -> Tuple[np.ndarray, np.ndarray, np.ndarray]: 17 | base_path = Path(__file__).parent 18 | folder_path = (base_path / "../building_energy_storage_simulation/data/preprocessed/").resolve() 19 | 20 | load = pd.read_csv(folder_path / 'electricity_load_profile.csv')[ 21 | 'Load [kWh]'] 22 | price = pd.read_csv(folder_path / 'electricity_price_profile.csv')[ 23 | 'Day Ahead Auction'] 24 | generation = pd.read_csv(folder_path / 'solar_generation_profile.csv')[ 25 | 'Generation [kWh]'] 26 | return np.array(load), np.array(price), np.array(generation) 27 | 28 | 29 | def plot_control_trajectory(residual_load, augmented_load, price, battery_power) -> None: 30 | ax = plt.subplot() 31 | ax.plot(residual_load, label='Residual Load') 32 | ax.plot(augmented_load, label='Augmented Load') 33 | ax.plot(price, '--', label='Price') 34 | ax.plot(battery_power, label='Battery Power') 35 | plt.ylabel('Load and Battery Power Applied (kW) & Price (Cent per kWh)') 36 | plt.xlabel('Time Step') 37 | ax.legend() 38 | ax.grid() 39 | plt.show() 40 | -------------------------------------------------------------------------------- /tests/test_building_simulation.py: -------------------------------------------------------------------------------- 1 | import pytest 2 | 3 | from building_energy_storage_simulation import BuildingSimulation 4 | 5 | 6 | def test_energy_consumption_is_trimmed_to_0(): 7 | sim = BuildingSimulation(electricity_load_profile=[0], solar_generation_profile=[100], electricity_price=[1]) 8 | # Don't charge the battery, meaning don't do anything with the 100kWh energy we gained from the solar system. 9 | electricity_consumption, excess_energy = sim.simulate_one_step(0) 10 | # Still the consumption is 0, as we loose excess electricity which we do not use to charge the battery. 11 | assert electricity_consumption == 0 12 | 13 | 14 | def test_simulation_reads_default_data_profiles(): 15 | sim = BuildingSimulation() 16 | assert len(sim.electricity_load_profile) == 8760 17 | assert len(sim.solar_generation_profile) == 8760 18 | 19 | 20 | @pytest.mark.parametrize( 21 | "electricity_price", [1, 1.1] 22 | ) 23 | def test_simulation_scalar_electricity_price_converted_into_profile(electricity_price): 24 | sim = BuildingSimulation(electricity_load_profile=[0, 0, 0], 25 | solar_generation_profile=[0, 0, 0], 26 | electricity_price=electricity_price) 27 | assert len(sim.electricity_price) == 3 28 | 29 | 30 | def test_simulation_throws_when_data_profiles_have_unequal_length(): 31 | with pytest.raises(Exception) as exception_info: 32 | BuildingSimulation(electricity_load_profile=[0, 0, 0], 33 | solar_generation_profile=[0, 0]) 34 | assert exception_info.errisinstance(ValueError) 35 | -------------------------------------------------------------------------------- /example_solutions/observation_wrapper.py: -------------------------------------------------------------------------------- 1 | import gymnasium 2 | import numpy as np 3 | 4 | 5 | class ObservationWrapper(gymnasium.Wrapper): 6 | """ 7 | Combines generation and load into one variable to reduce dimensionality of the observation space. 8 | """ 9 | def __init__(self, env, forecast_length): 10 | super().__init__(env) 11 | 12 | self.forecast_length = forecast_length 13 | original_observation_space_length = self.observation_space.shape[0] 14 | self.observation_space = gymnasium.spaces.Box(shape=(original_observation_space_length - forecast_length,), 15 | low=-np.inf, 16 | high=np.inf, dtype=np.float32) 17 | 18 | def reset(self, seed: int = 42, options=None): 19 | obs, info = self.env.reset() 20 | return self.convert_observation(obs), info 21 | 22 | def step(self, action): 23 | obs, reward, done, trunc, info = self.env.step(action) 24 | return self.convert_observation(obs), reward, done, trunc, info 25 | 26 | def convert_observation(self, obs): 27 | load_forecast = obs[1: self.forecast_length + 1] 28 | generation_forecast = obs[self.forecast_length + 1: 2 * self.forecast_length + 1] 29 | price_forecast = obs[2 * self.forecast_length + 1: 3 * self.forecast_length + 1] 30 | soc = obs[0] 31 | return np.concatenate(([soc], 32 | load_forecast - generation_forecast, 33 | price_forecast), 34 | axis=0) 35 | -------------------------------------------------------------------------------- /docs/source/index.rst: -------------------------------------------------------------------------------- 1 | .. building-energy-storage-simulation documentation master file, created by 2 | sphinx-quickstart on Wed Dec 21 13:41:14 2022. 3 | You can adapt this file completely to your liking, but it should at least 4 | contain the root `toctree` directive. 5 | 6 | Welcome to building-energy-storage-simulation's documentation! 7 | ============================================================== 8 | 9 | The `Building Energy Storage Simulation `_ serves as open source OpenAI gym (now 10 | `gymnasium `_) environment for reinforcement learning. The environment 11 | represents a building with an energy storage (in form of a battery) and a solar energy system. The aim is to control 12 | the energy storage in such a way that the energy of the solar system can be used optimally. 13 | 14 | A more detailed description of the task itself and the description of the markov decision process, containing 15 | information about the action space, observation space and reward can be found 16 | `here `_. 17 | 18 | Contents 19 | -------- 20 | 21 | .. toctree:: 22 | :maxdepth: 1 23 | :Caption: API 24 | 25 | modules/environment 26 | modules/simulation 27 | modules/battery 28 | 29 | Thanks To 30 | --------- 31 | 32 | The inspiration of this project and the data profiles come from the 33 | `CityLearn `_ environment. Anyhow, this project focuses on 34 | the ease of usage and the simplicity of its implementation. 35 | 36 | -------------------------------------------------------------------------------- /docs/imgs/overview.drawio: -------------------------------------------------------------------------------- 1 | 7VjRjps4FP2aPBIFAwl5nGQmbaXudqTZVbtPyAM3YI2xkW0myX792mCTBKgy3WYqzah5CPbxta8599jXZhKsy/0HgaviD54BnaBZtp8EtxOE/BBF+mGQQ4ss0KwFckEya3QEHsi/YEFnVpMM5Jmh4pwqUp2DKWcMUnWGYSH47txsy+m51wrnMAAeUkyH6FeSqaJFY7Q44h+B5IXz7M+XbUuJnbF9E1ngjO9OoOBuEqwF56otlfs1UEOe46Xtt/lOazcxAUy9pMMtf84//VltxV95+iX6WqjFc+HZ6DxjWtsXtpNVB8fAriAKHiqcmvpOR3kSrApVUl3zdRHLquV9S/agXa2kEvypIyvUiPUBQsH+u5P3O0q0loCXoMRBm7gOS8uilVHk9LE7BsWfW6w4DYgDsRVC3o195EoXLF0/QN38MnVKEMxyU1tdYHFImuAKK8KZrqLF7DosouCcxWAeDViMR0jsNHx1EoPLJOp1U5kiKZulesqaIYPotXpDSW6IUrw6QT/jR6D3XBJL4yNXipfagJqGFU6fcsFrlq055aLxFWybnzZpnN04ac/GdG7nc1soZfaiG0ME2qQZC6ZE70ZbwjIQ01R7RJsMK6wfBpf6CVLqiBFMPS2IJw0EhkYUeyusFIhDInXsa+n5KJ5WLL9G5P3e+oni4fpBY+vn1UK/eIehn10MfS4AmLfjgmZearzrAdHGLLqNrJknOcVCq4IB9VpTYCDyg1fROu/04PYLN/0MtrimeoarLWdqg0tCTZw/An0Gw4dtsNnVR7Y+0p1QOkSvsX/3dp4wGNm/UTymv/iV9Ld8h/p7wdbTygmzzKv4DoSX00NVeCj0qr3Zg0L9t9YHLMLyRBmDhNUpBSySimKmkqbTWVm2eaqtFbyW8IZ1GgX9c8YwQ/rBckSnDry6Tt3O/VuoA6HeUe1OkJSog5YfhQROgQN1qkxKLFWiBGayJFK2atXmtSK0M33Dqg1nPdWGQ9UuR5L78rVyu/+CewWw7MZc0HSNcWZUm2FZGNk0uj2RcKPApmH280GAPVHf7MCm/I8ZdurPF7Z+u7d+msrBVobnc9N8D4JowkBYq/YdIRtcKS8G8iRQ0UigHCaA6s32+Xz4sehZD/ecaMedToKop5M4ns6C81Ekr0UKtuPpdbI3Vtg/Uca9RK2wyEENBmr01L35T0jsBfev3xL75RIL474sltMYLY+//6e2KBwZ9pfqzVHxrtIwupiGd4BVobNvc1fxDK2RSTYdPvPfcN4MUO97SDxyK3G59Oy058CrZ06nqnclM/+izGriHT+K+Av3SeTvT8mdg5MvtdLXEkhQcjTFVeWZvu1h8ZFTZU+Lb1eU88EVZCjK+Ygm5z8uSV09fn9uN8rjV/zg7j8= -------------------------------------------------------------------------------- /example_solutions/deep_reinforcement_learning/train.py: -------------------------------------------------------------------------------- 1 | import os 2 | 3 | from stable_baselines3 import SAC 4 | from stable_baselines3.common.monitor import Monitor 5 | from stable_baselines3.common.vec_env import DummyVecEnv, VecNormalize 6 | 7 | from building_energy_storage_simulation import BuildingSimulation, Environment 8 | from example_solutions.helper import BATTERY_CAPACITY, BATTERY_POWER, TEST_INDEX_START, read_data 9 | from example_solutions.observation_wrapper import ObservationWrapper 10 | 11 | NUM_FORECAST_STEPS = 8 12 | RESULT_PATH = 'rl_example/' 13 | 14 | if __name__ == "__main__": 15 | os.makedirs(RESULT_PATH, exist_ok=True) 16 | 17 | load, price, generation = read_data() 18 | load_train = load[:TEST_INDEX_START] 19 | price_train = price[:TEST_INDEX_START] 20 | generation_train = generation[:TEST_INDEX_START] 21 | 22 | # Create Training Environment 23 | sim = BuildingSimulation(electricity_load_profile=load_train, 24 | solar_generation_profile=generation_train, 25 | electricity_price=price_train, 26 | max_battery_charge_per_timestep=BATTERY_POWER, 27 | battery_capacity=BATTERY_CAPACITY) 28 | 29 | env = Environment(sim, num_forecasting_steps=NUM_FORECAST_STEPS, max_timesteps=len(load_train) - NUM_FORECAST_STEPS) 30 | # ObservationWrapper combines forecast of load and generation to one residual load forecast 31 | env = ObservationWrapper(env, NUM_FORECAST_STEPS) 32 | initial_obs, info = env.reset() 33 | print(initial_obs) 34 | 35 | # Wrap with Monitor() so a log of the training is saved 36 | env = Monitor(env, filename=RESULT_PATH) 37 | # Warp with DummyVecEnc() so the observations and reward can be normalized using VecNormalize() 38 | env = DummyVecEnv([lambda: env]) 39 | env = VecNormalize(env, norm_obs=True, norm_reward=True) 40 | 41 | # Train :-) 42 | model = SAC("MlpPolicy", env, verbose=1, gamma=0.95) 43 | model.learn(total_timesteps=200_000) 44 | # Store the trained Model and environment stats (which are needed as we are standardizing the observations and 45 | # reward using VecNormalize()) 46 | model.save(RESULT_PATH + 'model') 47 | env.save(RESULT_PATH + 'env.pkl') -------------------------------------------------------------------------------- /example_solutions/optimal_control_problem.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import pyomo.environ as pyo 3 | from helper import BATTERY_CAPACITY, BATTERY_POWER, TEST_INDEX_END, TEST_INDEX_START, plot_control_trajectory, read_data 4 | 5 | 6 | def build_optimization_problem(residual_fixed_load, price, soc, battery_power, battery_capacity, delta_time_hours=1): 7 | time = range(len(residual_fixed_load)) 8 | soc_time = range(len(residual_fixed_load) + 1) 9 | max_power_charge = battery_power 10 | max_power_discharge = -1 * battery_power 11 | max_soc = 100 12 | min_soc = 0 13 | soc_init = soc 14 | energy_capacity = battery_capacity 15 | 16 | m = pyo.AbstractModel() 17 | m.power = pyo.Var(time, domain=pyo.Reals, bounds=(max_power_discharge, max_power_charge)) 18 | m.soc = pyo.Var(soc_time, bounds=(min_soc, max_soc)) 19 | 20 | def obj_expression(m): 21 | # pyo.log to make the objective expression smooth and therefore solvable 22 | return sum([price[i] * pyo.log(1 + pyo.exp(m.power[i] + residual_fixed_load[i])) for i in time]) 23 | 24 | m.OBJ = pyo.Objective(rule=obj_expression, sense=pyo.minimize) 25 | 26 | def soc_start_rule(m): 27 | return m.soc[0] == soc_init 28 | 29 | m.soc_start = pyo.Constraint(rule=soc_start_rule) 30 | 31 | def soc_constraint_rule(m, i): 32 | # Define the system dynamics as constraint 33 | return m.soc[i + 1] == float(100) * delta_time_hours * (m.power[i]) / energy_capacity + m.soc[i] 34 | 35 | m.soc_constraints = pyo.Constraint(time, rule=soc_constraint_rule) 36 | 37 | return m.create_instance() 38 | 39 | 40 | if __name__ == "__main__": 41 | solver = pyo.SolverFactory('ipopt') 42 | 43 | load, price, generation = read_data() 44 | 45 | load_eval = load[TEST_INDEX_START:TEST_INDEX_END] 46 | price_eval = price[TEST_INDEX_START:TEST_INDEX_END] 47 | generation_eval = generation[TEST_INDEX_START:TEST_INDEX_END] 48 | 49 | residual_fixed_load_eval = load_eval - generation_eval 50 | time = range(len(residual_fixed_load_eval)) 51 | 52 | m = build_optimization_problem(residual_fixed_load_eval, 53 | price_eval, 54 | soc=0, 55 | battery_power=BATTERY_POWER, 56 | battery_capacity=BATTERY_CAPACITY) 57 | solver.solve(m, tee=True) 58 | t = [time[i] for i in time] 59 | 60 | baseline_cost = sum(residual_fixed_load_eval[residual_fixed_load_eval > 0] * price_eval[residual_fixed_load_eval > 0]) 61 | augmented_load = residual_fixed_load_eval + np.array([(pyo.value(m.power[i])) for i in time]) 62 | cost = sum(augmented_load[augmented_load > 0] * price_eval[augmented_load > 0]) 63 | 64 | print('baseline cost: ' + str(baseline_cost)) 65 | print('cost: ' + str(cost)) 66 | print('savings in %: ' + str(1 - cost/baseline_cost)) 67 | 68 | plot_control_trajectory(residual_load=[(residual_fixed_load_eval[i]) for i in time], 69 | augmented_load=augmented_load, 70 | price=price_eval, 71 | battery_power=[(pyo.value(m.power[i])) for i in time] 72 | ) 73 | -------------------------------------------------------------------------------- /example_solutions/model_predictive_control.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import pyomo.environ as pyo 3 | from helper import BATTERY_CAPACITY, BATTERY_POWER, TEST_INDEX_END, TEST_INDEX_START, plot_control_trajectory, read_data 4 | from optimal_control_problem import build_optimization_problem 5 | 6 | from building_energy_storage_simulation import BuildingSimulation, Environment 7 | 8 | FORECAST_LENGTH = 24 9 | 10 | 11 | def normalize_to_minus_one_to_one(x, min_value, max_value): 12 | return -1 + 2 * (x - min_value) / (max_value - min_value) 13 | 14 | 15 | solver = pyo.SolverFactory('ipopt') 16 | 17 | load, price, generation = read_data() 18 | load_eval = load[TEST_INDEX_START:] 19 | price_eval = price[TEST_INDEX_START:] 20 | generation_eval = generation[TEST_INDEX_START:] 21 | 22 | num_eval_timesteps = TEST_INDEX_END - TEST_INDEX_START 23 | 24 | sim = BuildingSimulation(electricity_load_profile=load_eval, 25 | solar_generation_profile=generation_eval, 26 | electricity_price=price_eval, 27 | max_battery_charge_per_timestep=BATTERY_POWER, 28 | battery_capacity=BATTERY_CAPACITY) 29 | env = Environment(sim, num_forecasting_steps=FORECAST_LENGTH, max_timesteps=num_eval_timesteps) 30 | 31 | obs, info = env.reset() 32 | done = False 33 | 34 | actions, residual_loads, prices = (np.array([]), np.array([]), np.array([])) 35 | 36 | t = 0 37 | while not done: 38 | load_forecast = obs[1: FORECAST_LENGTH + 1] 39 | generation_forecast = obs[FORECAST_LENGTH + 1: 2 * FORECAST_LENGTH + 1] 40 | price_forecast = obs[2 * FORECAST_LENGTH + 1: 3 * FORECAST_LENGTH + 1] 41 | residual_load_forecast = load_forecast - generation_forecast 42 | soc = obs[0] 43 | 44 | optimization_problem = build_optimization_problem(residual_fixed_load=residual_load_forecast, 45 | price=price_forecast, 46 | # Convert SOC due to different SOC definitions 47 | soc=soc / BATTERY_CAPACITY * 100, 48 | battery_capacity=BATTERY_CAPACITY, 49 | battery_power=BATTERY_POWER) 50 | solver.solve(optimization_problem, tee=True) 51 | # Only apply the first action of the optimal solution in each iteration. This is a key concept of MPC. 52 | action = pyo.value(optimization_problem.power[0]) 53 | # Normalize action, as the environment expects normalized actions. 54 | normalized_action = normalize_to_minus_one_to_one(action, -1 * BATTERY_POWER, BATTERY_POWER) 55 | # Apply action to the environment and get new observation aka. state which is used to build the optimal control 56 | # problem of the next time step. 57 | obs, _, done, _, _ = env.step(normalized_action) 58 | 59 | residual_loads = np.append(residual_loads, residual_load_forecast[0]) 60 | prices = np.append(prices, price_forecast[0]) 61 | actions = np.append(actions, action) 62 | t += 1 63 | 64 | baseline_cost = sum(residual_loads[residual_loads > 0] * prices[residual_loads > 0]) 65 | augmented_load = residual_loads + actions 66 | cost = sum(augmented_load[augmented_load > 0] * prices[augmented_load > 0]) 67 | 68 | print('baseline cost: ' + str(baseline_cost)) 69 | print('cost: ' + str(cost)) 70 | print('savings in %: ' + str(1 - cost/baseline_cost)) 71 | 72 | plot_control_trajectory(residual_load=residual_loads, 73 | augmented_load=residual_loads + actions, 74 | price=prices, 75 | battery_power=actions) 76 | 77 | -------------------------------------------------------------------------------- /example_solutions/deep_reinforcement_learning/evaluate.py: -------------------------------------------------------------------------------- 1 | # Plot the training process 2 | import pandas as pd 3 | from stable_baselines3 import SAC 4 | from stable_baselines3.common.results_plotter import plot_results 5 | from stable_baselines3.common.vec_env import DummyVecEnv, VecNormalize 6 | 7 | from building_energy_storage_simulation import BuildingSimulation, Environment 8 | from example_solutions.deep_reinforcement_learning.train import NUM_FORECAST_STEPS, RESULT_PATH 9 | from example_solutions.helper import ( 10 | BATTERY_CAPACITY, 11 | BATTERY_POWER, 12 | TEST_INDEX_END, 13 | TEST_INDEX_START, 14 | plot_control_trajectory, 15 | read_data, 16 | ) 17 | from example_solutions.observation_wrapper import ObservationWrapper 18 | 19 | 20 | def evaluate(env, agent=None): 21 | # Do the evaluation 22 | actions, observations, electricity_consumption, price, rewards = ([], [], [], [], []) 23 | done = False 24 | obs = env.reset() 25 | while not done: 26 | if agent is None: # noqa 27 | action = [[0]] 28 | else: 29 | action = [agent.predict(obs, deterministic=True)[0][0]] 30 | 31 | obs, r, done, info = env.step([action[0][0]]) 32 | 33 | actions.append(action[0][0]) 34 | original_obs = env.get_original_obs()[0] 35 | observations.append(original_obs) 36 | electricity_consumption.append(info[0]['electricity_consumption']) 37 | price.append(info[0]['electricity_price']) 38 | rewards.append(r) 39 | 40 | return pd.DataFrame({ 41 | 'action': actions, 42 | 'observations': observations, 43 | 'electricity_consumption': electricity_consumption, 44 | 'electricity_price': price, 45 | 'reward': rewards 46 | }) 47 | 48 | 49 | if __name__ == "__main__": 50 | # Plot evolution of reward during training 51 | try: 52 | plot_results(RESULT_PATH, x_axis='timesteps', task_name='title', num_timesteps=None) 53 | except: # noqa 54 | print('Training Reward Plot could not be created') 55 | 56 | load, price, generation = read_data() 57 | load_eval = load[TEST_INDEX_START:] 58 | price_eval = price[TEST_INDEX_START:] 59 | generation_eval = generation[TEST_INDEX_START:] 60 | 61 | num_eval_timesteps = TEST_INDEX_END - TEST_INDEX_START 62 | 63 | eval_sim = BuildingSimulation(electricity_load_profile=load_eval, 64 | solar_generation_profile=generation_eval, 65 | electricity_price=price_eval, 66 | max_battery_charge_per_timestep=BATTERY_POWER, 67 | battery_capacity=BATTERY_CAPACITY) 68 | 69 | eval_env = Environment(eval_sim, num_forecasting_steps=NUM_FORECAST_STEPS, max_timesteps=num_eval_timesteps) 70 | eval_env = ObservationWrapper(eval_env, NUM_FORECAST_STEPS) 71 | eval_env = DummyVecEnv([lambda: eval_env]) 72 | # It is important to load the environmental statistics here as we use a rolling mean calculation ! 73 | eval_env = VecNormalize.load(RESULT_PATH + 'env.pkl', eval_env) 74 | eval_env.training = False 75 | 76 | model = SAC.load(RESULT_PATH + 'model') 77 | 78 | trajectory = evaluate(eval_env, model) 79 | baseline_trajectory = evaluate(eval_env, None) 80 | 81 | cost = sum(trajectory['electricity_price'] * trajectory['electricity_consumption']) 82 | baseline_cost = sum(baseline_trajectory['electricity_price'] * baseline_trajectory['electricity_consumption']) 83 | 84 | print('baseline cost: ' + str(baseline_cost)) 85 | print('cost: ' + str(cost)) 86 | print('savings in %: ' + str(1 - cost / baseline_cost)) 87 | 88 | observation_df = trajectory['observations'].apply(pd.Series) 89 | augmented_load = observation_df[1] + trajectory['action'] * BATTERY_POWER 90 | plot_control_trajectory(residual_load=observation_df[1], 91 | augmented_load=augmented_load, 92 | price=trajectory['electricity_price'], 93 | battery_power=trajectory['action'] * BATTERY_POWER) 94 | -------------------------------------------------------------------------------- /tests/test_environment.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import pytest 3 | 4 | from building_energy_storage_simulation import Environment 5 | from building_energy_storage_simulation.building_simulation import BuildingSimulation 6 | 7 | 8 | @pytest.fixture(scope='module') 9 | def building_simulation(): 10 | return BuildingSimulation(electricity_load_profile=[1, 2, 3, 4, 5, 6, 7, 8, 9, 10], 11 | solar_generation_profile=[10, 9, 8, 7, 6, 5, 4, 3, 2, 1], 12 | electricity_price=[1, 2, 3, 4, 5, 6, 7, 8, 9, 10], 13 | max_battery_charge_per_timestep=20) 14 | 15 | 16 | def test_environment_noop_step(building_simulation): 17 | """ 18 | Perfect forecast of energy consumption of time step t+1 equals the actual energy consumption of that time step 19 | if the energy is not charged or discharged. 20 | """ 21 | env = Environment(building_simulation=building_simulation, num_forecasting_steps=3, max_timesteps=6) 22 | initial_obs = env.reset() 23 | obs = env.step(0) 24 | assert initial_obs[0][2] == obs[0][1] 25 | 26 | 27 | @pytest.mark.parametrize( 28 | "data_profile_length, num_forecasting_steps", [(2, 1), (9, 0)] 29 | ) 30 | def test_terminated_at_timelimit_reached(data_profile_length, num_forecasting_steps): 31 | dummy_profile = np.zeros(data_profile_length) 32 | building_sim = BuildingSimulation(electricity_price=dummy_profile, 33 | solar_generation_profile=dummy_profile, 34 | electricity_load_profile=dummy_profile) 35 | env = Environment(building_simulation=building_sim, 36 | num_forecasting_steps=num_forecasting_steps, 37 | max_timesteps=data_profile_length - num_forecasting_steps) 38 | env.reset() 39 | print(range(data_profile_length - num_forecasting_steps)) 40 | for _ in range(data_profile_length - num_forecasting_steps): 41 | obs, reward, terminated, trunc, info = env.step(0) 42 | assert terminated is True 43 | 44 | 45 | def test_observation_size(building_simulation): 46 | env = Environment(building_simulation=building_simulation, max_timesteps=5, num_forecasting_steps=4) 47 | initial_obs, info = env.reset() 48 | obs, reward, terminated, trunc, info = env.step(0) 49 | assert len(obs) == 13 50 | 51 | 52 | def test_initial_obs_step_obs_same_size(building_simulation): 53 | env = Environment(building_simulation=building_simulation, max_timesteps=5, num_forecasting_steps=4) 54 | initial_obs, info = env.reset() 55 | obs, reward, terminated, trunc, info = env.step(0) 56 | assert len(obs) == len(initial_obs) 57 | 58 | 59 | def test_max_battery_charge_per_timestep(building_simulation): 60 | env = Environment(building_simulation=building_simulation, max_timesteps=5, num_forecasting_steps=4) 61 | initial_obs, info = env.reset() 62 | obs, reward, terminated, trunc, info = env.step(1) 63 | # Position 0 in observation is the state_of_charge. Check, if the state of charge increased by MAX_BATTERY_CHARGE 64 | # when fully charging 65 | assert obs[0] == initial_obs[0] + 20 66 | 67 | 68 | def test_reset(building_simulation): 69 | env = Environment(building_simulation=building_simulation, max_timesteps=5, num_forecasting_steps=4) 70 | env.reset() 71 | env.step(1) 72 | env.reset() 73 | assert env.building_simulation.step_count == 0 74 | assert env.building_simulation.battery.state_of_charge == 0 75 | 76 | 77 | def test_default_initialization_runs_without_throwing(): 78 | sim = BuildingSimulation() 79 | env = Environment(sim) 80 | env.reset() 81 | env.step(1) 82 | assert env.building_simulation.step_count == 1 83 | 84 | 85 | def test_set_random_first_time_step_always_0_for_data_profile_length_2(): 86 | dummy_profile = [0, 0] 87 | sim = BuildingSimulation(electricity_price=dummy_profile, 88 | electricity_load_profile=dummy_profile, 89 | solar_generation_profile=dummy_profile) 90 | env = Environment(sim, randomize_start_time_step=True, max_timesteps=1, num_forecasting_steps=1) 91 | env.reset() 92 | assert env.building_simulation.start_index == 0 93 | 94 | 95 | def test_set_random_first_time_step(): 96 | dummy_profile = np.zeros(1000) 97 | sim = BuildingSimulation(electricity_price=dummy_profile, 98 | electricity_load_profile=dummy_profile, 99 | solar_generation_profile=dummy_profile) 100 | env = Environment(sim, randomize_start_time_step=True, max_timesteps=1, num_forecasting_steps=1) 101 | env.reset() 102 | # This test is very unlikely to fail ;) 103 | assert env.building_simulation.start_index != 0 104 | 105 | 106 | @pytest.mark.parametrize( 107 | "reset", [True, False] 108 | ) 109 | def test_forecasts_are_randomized_in_observation(reset): 110 | dummy_profile = np.zeros(10) 111 | building_sim = BuildingSimulation(electricity_price=dummy_profile, 112 | solar_generation_profile=dummy_profile, 113 | electricity_load_profile=dummy_profile) 114 | env = Environment(building_simulation=building_sim, 115 | num_forecasting_steps=4, 116 | max_timesteps=6, 117 | randomize_forecasts_in_observation=True) 118 | 119 | if reset: 120 | obs, _ = env.reset() 121 | else: 122 | env.reset() 123 | obs, _, _, _, _ = env.step(0) 124 | 125 | load_forecast = np.array(obs[1:5]) 126 | generation_forecast = obs[5:9] 127 | price_forecast = obs[9:14] 128 | assert not np.array_equal(generation_forecast, load_forecast) 129 | assert not np.array_equal(price_forecast, load_forecast) 130 | assert not np.array_equal(price_forecast, dummy_profile) 131 | 132 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Building Energy Storage Simulation 2 | 3 | isolated 4 | 5 | The Building Energy Storage Simulation serves as an OpenAI gym (now [gymnasium](https://github.com/Farama-Foundation/Gymnasium)) environment 6 | for Reinforcement Learning. The environment represents a building with an energy storage (in the form of a battery) and a 7 | solar energy system. The building is connected to a power grid with time-varying electricity prices. The task is to 8 | control the energy storage so that the total cost of electricity is minimized. 9 | 10 | The inspiration for this project and the data profiles come from the [CityLearn](https://github.com/intelligent-environments-lab/CityLearn) environment. Anyhow, this project focuses on the ease of usage and the simplicity of its implementation. Therefore, this project serves as a playground for those who want to get started with reinforcement learning for energy management system control. 11 | 12 | ## Installation 13 | 14 | By using pip just: 15 | 16 | ``` 17 | pip install building-energy-storage-simulation 18 | ``` 19 | 20 | or if you want to continue developing the package: 21 | 22 | ``` 23 | git clone https://github.com/tobirohrer/building-energy-storage-simulation.git && cd building-energy-storage-simulation 24 | pip install -e .[dev] 25 | ``` 26 | 27 | ## Usage 28 | 29 | ```python 30 | from building_energy_storage_simulation import Environment, BuildingSimulation 31 | 32 | simulation = BuildingSimulation() 33 | env = Environment(building_simulation=simulation) 34 | 35 | env.reset() 36 | env.step(1) 37 | ... 38 | ``` 39 | 40 | **Important note:** This environment is implemented by using [gymnasium](https://github.com/Farama-Foundation/Gymnasium) (the proceeder of OpenAI gym). Meaning, if you are using a reinforcement learning library like [Stable Baselines3](https://github.com/DLR-RM/stable-baselines3) make sure it supports [gymnasium](https://github.com/Farama-Foundation/Gymnasium) environments. 41 | 42 | ## Task Description 43 | 44 | The simulation contains a building with an energy load profile attached to it. The load is always automatically covered by 45 | 46 | - primarily using electricity generated by the solar energy system, 47 | - and secondary by using the remaining required electricity "from the grid" 48 | 49 | When energy is taken from the grid, costs are incurred that can vary depending on the time (if a price profile is passed 50 | as `electricity_price` to `BuildingSimulation`). The simulated building contains a battery that be controlled by 51 | **charging** and **discharging** energy. The goal is to find control strategies to optimize the use of energy storage 52 | by e.g. charging whenever electricity prices are high or whenever there is a surplus of solar generation. It is important 53 | to note that no energy can be fed into the grid. This means any surplus of solar energy which is not used to charge the 54 | battery is considered lost. 55 | 56 | ### Reward 57 | 58 | $$r_t = -1 * electricity\\_consumed_t * electricity\\_price_t $$ 59 | 60 | Note, that the term `electricity_consumed` cannot be negative. This means excess energy from the solar 61 | energy system which is not consumed by the electricity load or by charging the battery is considered lost 62 | (`electricity_consumed` is 0 in this case). 63 | 64 | ### Action Space 65 | 66 | | Action | Min | Max | 67 | |----------|----------|--------| 68 | | Charge | -1 | 1 | 69 | 70 | The actions lie in the interval of [-1;1]. The action represents a fraction of the maximum energy that can be retrieved from the battery (or used to charge the battery) per time step. 71 | 72 | - 1 means maximum charging the battery. The maximum charge per time step is defined by the parameter `max_battery_charge_per_timestep`. 73 | - -1 means maximum discharging the battery, meaning "gaining" electricity out of the battery 74 | - 0 means don't charge or discharge 75 | 76 | ### Observation Space 77 | 78 | | Index | Observation | Min | Max | 79 | |-------------|---------------------------------------|---------------------------|------------------------------| 80 | | 0 | State of Charge (in kWh) | 0 | `battery_capacity` | 81 | | [1; n] | Forecast Electric Load (in kWh) | Min of Load Profile | Max of Load Profile | 82 | | [n+1; 2*n] | Forecast Solar Generation (in kWh) | Min of Generation Profile | Max of Generation Profile | 83 | | [2n+1; 3*n] | Electricity Price (in € cent per kWh) | Min of Price Profile | Max of Price Profile | 84 | 85 | 86 | The length of the observation depends on the length of the forecast ($n$) used. By default, the simulation uses a forecast length of 4. 87 | This means 4 time steps of an electric load forecast, 4 time steps of a solar generation forecast, and 4 time steps of the 88 | electric price profiles are included in the observation. 89 | In addition to that, the information about the current state of charge of the battery is contained in the observation. 90 | 91 | The length of the forecast can be defined by setting the parameter `num_forecasting_steps` of the `Environment()`. 92 | 93 | 94 | ### Episode Ends 95 | 96 | The episode ends if the `max_timesteps` of the `Environment()` are reached. 97 | 98 | ## Example Solutions 99 | 100 | The folder [example_solutions](example_solutions) contains three different example solutions to solve the problem 101 | described. 102 | 103 | 1. By applying deep reinforcement learning using the framework [stable-baselines3](https://github.com/DLR-RM/stable-baselines3). 104 | 2. By formulating the problem as an optimal control problem (OCP) using [pyomo](http://www.pyomo.org/). In this case, it 105 | is assumed that the forecast for the price, load, and generation data for the whole period is available. 106 | 3. By model predictive control, which solves the optimal control problem formulation from 2. in each time step in a closed loop manner. 107 | In contrast to 2. only a forecast of a fixed length is given in each iteration. 108 | 109 | Note that the execution of the example solutions requires additional dependencies which are not specified inside `setup.py`. 110 | Therefore, make sure to install the required Python packages defined in `requirements.txt`. Additionally, an installation 111 | of the `ipopt` solver is required to solve the optimal control problem 112 | (by using conda, simply run `conda install -c conda-forge ipopt`). 113 | 114 | ## Code Documentation 115 | 116 | The documentation is available at [https://building-energy-storage-simulation.readthedocs.io/](https://building-energy-storage-simulation.readthedocs.io/en/master/) 117 | 118 | ## Contribute & Contact 119 | 120 | As I just started with this project, I am very happy for any kind of 121 | contribution! In case you want to contribute, or if you have any 122 | questions, contact me via 123 | [discord](https://discord.com/users/tobirohrer#8654). 124 | 125 | --------------------------------------------------------------------------------