[Feature] Support named configs#1588
Open
ferenc-hechler wants to merge 15 commits intoPortkey-AI:mainfrom
Open
[Feature] Support named configs#1588ferenc-hechler wants to merge 15 commits intoPortkey-AI:mainfrom
ferenc-hechler wants to merge 15 commits intoPortkey-AI:mainfrom
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description:
Currently configs have to be provided in plain text as header parameter "x-portkey-config" in the client request.
This makes the gateway not usable standalone.
One main goal of the gateway is to hide the configuration details, especially secrets, from the clients.
One proposal often given is to add a reverse proxy like nginx, which can then add the plain text config,
but this adds another component, which has to be deployed and maintained.
This PR adds the possibility to provide a named_configs.json file on startup time,
which defines multiple named configurations and the client has to use the name of the configuration
instead of providing the full configuration text in the header parameter "x-portkey-config".
Also there is a possibility to define a configuration named "default"
which is used if there is no "x-portkey-config" header parameter.
An example configuration for ./named_configs.json:
To use named configurations the environment variable NAMED_CONFIGS has to be set.
$NAMED_CONFIGS can contain the JSON directly or a filename which contains the JSON, e.g. "./named_config.json".
Environment variables in the JSON values are resolved if available. Unresolved variables remain unchanged.
If the environment variable NAMED_CONFIGS is not set,
the gateway behaves in the same way as before.
But if the environment variable is set only config names can be provided in "x-portkey-config".
For invalid config names are handled as if no x-portkey-config parameter was given.
Tests Run/Test cases added: (required)
I am struggeling getting the tests and the server running at the same time.
The problem seems to be related to "await import" in env.ts,
which is only allowed in ESM (server), while "npx jest" runs with JCS and needs "requires".
Maybe someone can give me a hint how to get both working at the same time.
Manual Tests:
Using the ./named_config.js from above and start the docker container with:
In the startup logs there is a new line
Sending a curl request:
Currently there is debug logging showing the config resolution, so we can see, which config is used:
This means, the request is internally handled as if
{"provider":"openai","api_key":"Your OpenAI API Key"}was given in x-portkey-config.Succesfully tested the communication with a locally started Ollama server.
Type of Change: