Skip to content

develop.md

Latest commit

 

History

History
378 lines (296 loc) · 16.1 KB

develop.md

File metadata and controls

378 lines (296 loc) · 16.1 KB

EMHASS Development

When contributing to this repository, please first discuss the change you wish to make via issue, email, or any other method with the owners of this repository before making a change.

Please note we have a code of conduct, please follow it in all your interactions with the project.

Setup

There are multiple different approaches to developing for EMHASS.
The choice depends on your and preference (Python venv/DevContainer/Docker).
Below are some development workflow examples:
Note: It is preferred to run the actions and unittest once before submitting and pull request.

Step 1 - Fork

With your preferred Git tool of choice:
Fork the EMHASS github repository into your own account, then clone the forked repository into your local development platform. (ie. PC or Codespace) Here you may also wish to add the add the original/upstream repository as a remote, allowing you to fetch and merge new updates from the original repository.

A command example may be:

# on GitHub, Fork url, then:
git clone git@github.com:<YOURUSERNAME>/emhass.git
cd emhass
# add remote, call it upstream
git remote add upstream https://github.com/davidusb-geek/emhass.git

Step 2 - Develop

To develop and test code choose one of the following methods:

Method 1 - Python Virtual Environment

We can use python virtual environments to build, develop and test/unittest the code.

confirm terminal is in the root emhass directory before starting

Create a developer environment:

Using the uv package manager:

# With the 'test' packages to run unit tests locally.
uv sync --extra test
# If on ARM, try adding piwheels as an index.
#uv sync --extra test --index=https://www.piwheels.org/simple

Using virtualenv and pip:

virtualenv .venv

# Then activate the virtualenv, see below...

# With the 'test' packages to run unit tests locally.
python3 -m pip install -e '.[test]'

To activate the virtualenv, created by either uv or pip:

  • Linux: 
    source .venv/bin/activate
  • windows: 
    .venv\Scripts\activate.bat

This installs dependencies and creates a .venv virtualenv in the working directory. An IDE like VSCode should automatically catch that a new virtual env was created.

Set paths with environment variables:

  • Linux

    export OPTIONS_PATH="${PWD}/options.json" && export USE_OPTIONS="True" ##optional to test options.json
export CONFIG_PATH="${PWD}/config.yaml"
export SECRETS_PATH="${PWD}/secrets_emhass.yaml" ##optional to test secrets_emhass.yaml
export DATA_PATH="${PWD}/data/"

    Optionally, use direnv to have these variables handled for you.

  • windows

    set "OPTIONS_PATH=%cd%/options.json"  & ::  optional to test options.json
set "USE_OPTIONS=True"                & ::  optional to test options.json
set "CONFIG_PATH=%cd%/config.json"
set "SECRETS_PATH=%cd%/secrets_emhass.yaml" & ::  optional to test secrets_emhass.yam
set "DATA_PATH=%cd%/data/"

Make sure secrets_emhass.yaml has been created and set. Copy secrets_emhass(example).yaml for an example.

Run EMHASS

python3 ./src/emhass/web_server.py

or

emhass --action 'dayahead-optim' --config ./config.json --root ./src/emhass --costfun 'profit' --data ./data

Run unittests

pytest

Method 2: VS-Code Debug and Run via Dev Container

In VS-Code, you can run a Docker Dev Container to set up a virtual environment. The Dev Container's Container will be almost identical to the one build for EMHASS (Docker/Add-on). There you can edit and test EMHASS.

The recommended steps to run are:

  • Open forked root (emhass) folder inside of VS-Code
  • VS-Code will ask if you want to run in a dev-container, say yes (Dev Container must be set up first). (Shortcut: F1 > Dev Containers: Rebuild and Reopen in Container)
  • Edit some code...
  • Compile emhass by pressing control+shift+p > Tasks: Run Task > EMHASS Install. This has been set up in the tasks.json file. - Before run & debug, re-run EMHASS Install task every time a change has been made to emhass.
  • Launch and debug the program via the Run and Debug tab /Ctrl+Shift+D > EMHASS run This has been set up in the Launch.json .

Simulate Docker Method or Add-on method

Since the main difference between the two methods are how secrets are passed. You can switch between the two methods by:

Docker:

  • Create a secrets_emhass.yaml file and append your secret parameters

Add-on:

  • Modify the options.json file to contain you secret parameters

Unittests

Lastly, you can run all the unittests by heading to the Testing tab on the left hand side. This is recommended before creating a pull request.

Method 3 - Docker Virtual Environment

With Docker, you can test the production EMHASS environment for both Docker and Add-on methods.

Depending on the method you wish to test, the docker run command will require different passed arguments to function. See following examples:

Note: Make sure your terminal is in the root emhass repository directory before running the docker build.

Docker run Add-on Method:

docker build -t emhass/test .

# pass secrets via options.json (similar to what Home Assistant automatically creates from the addon configuration page)
docker run -it -p 5000:5000 --name emhass-test -v ./options.json:/data/options.json emhass/test

Note:

  • to apply a file change in the local EMHASS repository, you will need to re-build and re-run the Docker image/container in order for the change to take effect. (excluding volume mounted (-v) files/folders)
  • if you are planning to modify the configs: options.json, secrets_emhass.yaml or config.json, you can volume mount them with -v. This syncs the Host file to the file inside the container. If running inside of podman, add :z at the end of the volume mount E.g:-v ./options.json:/data/options.json:z

Docker run for Docker Method:

docker build -t emhass/test .

# pass the secrets_emhass.yaml
docker run -it -p 5000:5000 --name emhass-test -v ./secrets_emhass.yaml:/app/secrets_emhass.yaml emhass/test

Sync with local data folder

For those who wish to mount/sync the local data folder with the data folder from inside the docker container, volume mount the data folder with -v .

docker run ... -v ./data/:/data/ ...

You can also mount data files (ex .csv) separately

docker run... -v ./data/heating_prediction.csv:/data/ ...

Issue with TARGETARCH

If your docker build fails with an error related to TARGETARCH. It may be best to add your device's architecture manually:

Example with armhf architecture

docker build ... --build-arg TARGETARCH=armhf --build-arg os_version=raspbian ...

For armhf only, also pass a build-arg for os_version=raspbian

Delete built Docker image

We can delete the Docker image and container via:

# force delete Docker container
docker rm -f emhass-test 

# delete Docker image
docker rmi emhass/test

Other Docker Options

Rapid Testing
As editing and testing EMHASS via docker may be repetitive (rebuilding image and deleting containers), you may want to simplify the removal, build and run process.

For rapid Docker testing, try a command chain:
Linux:

docker build -t emhass/test . && docker run --rm -it -p 5000:5000 -v ./secrets_emhass.yaml:/app/secrets_emhass.yaml --name emhass-test emhass/test

The example command chain rebuilds the Docker image, and runs a new container with the newly built image. The --rm has been added to the docker run to delete the container once ended to avoid manual deletion every time.
This use case may not require any volume mounts (unless you use secrets_emhass.yaml) as the Docker build process will pull the latest configs as it builds.

Environment Variables
you can also pass location, key and url secret parameters via environment variables.

docker build -t emhass/test --build-arg build_version=addon-local .

docker run -it -p 5000:5000 --name emhass-test -e URL="YOURHAURLHERE" -e KEY="YOURHAKEYHERE" -e LAT="45.83" -e LON="6.86" -e ALT="4807.8" -e TIME_ZONE="Europe/Paris" emhass/test

This allows the user to set variables before the build Linux:

export EMHASS_URL="YOURHAURLHERE"
export EMHASS_KEY="YOURHAKEYHERE"
export TIME_ZONE="Europe/Paris"
export LAT="45.83"
export LON="6.86"
export ALT="4807.8"

docker build -t emhass/test --build-arg build_version=addon-local .

docker run -it -p 5000:5000 --name emhass-test -e EMHASS_KEY -e EMHASS_URL -e TIME_ZONE -e LAT -e LON -e ALT emhass/test

Example Docker testing pipeline

The following pipeline will run unittest and most of the EMHASS actions. This may be a good option for those who wish to test their changes against the production EMHASS environment.

Linux:
Assuming docker and git installed

#setup environment variables for test
export repo=https://github.com/davidusb-geek/emhass.git
export branch=master
#Ex. HAURL=https://localhost:8123/
export HAURL=HOMEASSISTANTURLHERE
export HAKEY=HOMEASSISTANTKEYHERE

git clone $repo
cd emhass 
git checkout $branch
# testing with option.json (replace -v options.json with secrets_emhass.yaml to test both secret files)
docker build -t emhass/test .
docker run --rm -it -p 5000:5000 --name emhass-test -v $(pwd)/data/heating_prediction.csv:/data/heating_prediction.csv -v $(pwd)/options.json:/app/options.json emhass/test
# run actions one-by-one, on a separate terminal
curl -i -H 'Content-Type:application/json' -X POST -d '{"pv_power_forecast":[0, 70, 141.22, 246.18, 513.5, 753.27, 1049.89, 1797.93, 1697.3, 3078.93], "prediction_horizon":10, "soc_init":0.5,"soc_final":0.6}' http://localhost:5000/action/naive-mpc-optim
curl -i -H 'Content-Type:application/json' -X POST -d {} http://localhost:5000/action/perfect-optim
curl -i -H 'Content-Type:application/json' -X POST -d {} http://localhost:5000/action/dayahead-optim
curl -i -H 'Content-Type:application/json' -X POST -d {} http://localhost:5000/action/forecast-model-fit
curl -i -H 'Content-Type:application/json' -X POST -d {} http://localhost:5000/action/forecast-model-predict
curl -i -H 'Content-Type:application/json' -X POST -d {} http://localhost:5000/action/forecast-model-tune
curl -i -H "Content-Type:application/json" -X POST -d  '{"csv_file": "heating_prediction.csv", "features": ["degreeday", "solar"], "target": "hour", "regression_model": "RandomForestRegression", "model_type": "heating_hours_degreeday", "timestamp": "timestamp", "date_features": ["month", "day_of_week"], "new_values": [12.79, 4.766, 1, 2] }' http://localhost:5000/action/regressor-model-fit
curl -i -H "Content-Type:application/json" -X POST -d  '{"mlr_predict_entity_id": "sensor.mlr_predict", "mlr_predict_unit_of_measurement": "h", "mlr_predict_friendly_name": "mlr predictor", "new_values": [8.2, 7.23, 2, 6], "model_type": "heating_hours_degreeday" }' http://localhost:5000/action/regressor-model-predict
curl -i -H 'Content-Type:application/json' -X POST -d {} http://localhost:5000/action/publish-data
# testing unittest (add extra necessary files via volume mount)
docker run --rm -it -p 5000:5000 --name emhass-test -v $(pwd)/tests/:/app/tests/ -v $(pwd)/data/:/data/ -v $(pwd)/"secrets_emhass(example).yaml":/app/"secrets_emhass(example).yaml" -v $(pwd)/options.json:/app/options.json -v $(pwd)/config_emhass.yaml:/app/config_emhass.yaml -v $(pwd)/secrets_emhass.yaml:/app/secrets_emhass.yaml emhass/test
# run unittest's on separate terminal after installing requests-mock
docker exec emhass-test apt-get update 
docker exec emhass-test apt-get install python3-requests-mock -y
docker exec emhass-test python3 -m unittest discover -s ./tests -p 'test_*.py' | grep error

Note: may need to set --build-arg TARGETARCH=YOUR-ARCH in docker build

User may wish to re-test with tweaked parameters such as lp_solver, weather_forecast_method and load_forecast_method, in config.json to broaden the testing scope. See Differences for more information on how the different methods of running EMHASS differ.

Adding a parameter

When enhancing EMHASS, users may like to add or modify the EMHASS parameters. To add a new parameter see the following steps:

Example parameter = this_parameter_is_amazing

  1. Append a line into associations.csv : So that build_params() knows what config catagorie to allocate the parameter
...
retrieve_hass_conf,,this_parameter_is_amazing
  • Alternatively if you want to support this parameter with the yaml conversion (Ie. allow the parameter to be converted from config_emhass.yaml) 
    ...
retrieve_hass_conf,his_parameter_is_amazing,this_parameter_is_amazing
  1. Append a line into the config_defaults.json To set a default value for the user if none is provided in config.json
"...": "...",
  "this_parameter_is_amazing": [
    0.1,
    0.1
  ]
  1. Update the Web UI in param_definitions.json To support the configuration website to generate the parameter in the list view, append the param_definitions.json file:
"this_parameter_is_amazing": {
      "friendly_name": "This parameter is amazing",
      "Description": "This parameter functions as you expect. It makes EMHASS AMAZING!",
      "input": "array.float",
      "default_value": 0.777
    }

Note: The default_value in this case acts (or should act) as last resort fallback if default_config.json is not found. It also acts as the default value when you append (press plus) to an array.* parameter

Screenshot from 2024-09-09 16-45-32

If you are only adding another option for a existing parameter, editing param_definitions.json file should be all you need (allowing the user to select the option from the configuration page):

"load_forecast_method": {
  "friendly_name": "Load forecast method",
  "Description": "The load forecast method that will be used. The options are ‘csv’ to load a CSV file or ‘naive’ for a simple 1-day persistence model.",
  "input": "select",
  "select_options": [
    "naive",
    "mlforecaster",
    "csv",
    "CALL_NEW_OPTION"
  ],
  "default_value": "naive"
},
  1. Update the Optimization Cache Key (command_line.py) If your new parameter affects the mathematical structure of the optimization problem (e.g., adding constraints, changing binary variables, or adding penalty weights), it must trigger a cache miss when changed.

Add your parameter to the OptimizationCacheKey dataclass and the _compute_cache_key method inside command_line.py:

@dataclass(frozen=True)
class OptimizationCacheKey:
    # ... existing parameters ...
    this_parameter_is_amazing: tuple

# Inside _compute_cache_key:
return OptimizationCacheKey(
    # ... existing parameters ...
    this_parameter_is_amazing=to_tuple(
        optim_conf.get("this_parameter_is_amazing", [])
    ),
)

Note on Deferrable Load Lists

If your parameter is a list that must match the number_of_deferrable_loads (like set_deferrable_max_startups), you must also ensure it gets padded correctly in utils.py. Add a call to check_def_loads() inside the build_params() method:

optim_conf["this_parameter_is_amazing"] = check_def_loads(
    optim_conf["number_of_deferrable_loads"],
    optim_conf,
    0, # Your default pad value
    "this_parameter_is_amazing",
    logger,
)

Step 3 - Pull request

Once developed, commit your code, and push the commit to your fork on Github. Once ready, submit a pull request with your fork to the davidusb-geek/emhass@master repository.