Skip to content

Commit 4962e7a

Browse files
OCA-git-botOCA-git-bot
committed
Merge PR #3399 into 19.0
Signed-off-by thomaspaulb
2 parents f0615ae + 06cb55f commit 4962e7a

File tree

13 files changed

+917
-0
lines changed

13 files changed

+917
-0
lines changed

‎bus_alt_connection/README.rst‎

Lines changed: 189 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,189 @@
1+
.. image:: https://odoo-community.org/readme-banner-image
2+
:target: https://odoo-community.org/get-involved?utm_source=readme
3+
:alt: Odoo Community Association
4+
5+
==================
6+
Bus Alt Connection
7+
==================
8+
9+
..
10+
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
11+
!! This file is generated by oca-gen-addon-readme !!
12+
!! changes will be overwritten. !!
13+
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
14+
!! source digest: sha256:ab7e1b9d5721f8cb27f93c58ed77e5034bc0099105ac6d5097f1bdc74a4e6973
15+
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
16+
17+
.. |badge1| image:: https://img.shields.io/badge/maturity-Beta-yellow.png
18+
:target: https://odoo-community.org/page/development-status
19+
:alt: Beta
20+
.. |badge2| image:: https://img.shields.io/badge/license-AGPL--3-blue.png
21+
:target: http://www.gnu.org/licenses/agpl-3.0-standalone.html
22+
:alt: License: AGPL-3
23+
.. |badge3| image:: https://img.shields.io/badge/github-OCA%2Fserver--tools-lightgray.png?logo=github
24+
:target: https://github.com/OCA/server-tools/tree/19.0/bus_alt_connection
25+
:alt: OCA/server-tools
26+
.. |badge4| image:: https://img.shields.io/badge/weblate-Translate%20me-F47D42.png
27+
:target: https://translation.odoo-community.org/projects/server-tools-19-0/server-tools-19-0-bus_alt_connection
28+
:alt: Translate me on Weblate
29+
.. |badge5| image:: https://img.shields.io/badge/runboat-Try%20me-875A7B.png
30+
:target: https://runboat.odoo-community.org/builds?repo=OCA/server-tools&target_branch=19.0
31+
:alt: Try me on Runboat
32+
33+
|badge1| |badge2| |badge3| |badge4| |badge5|
34+
35+
This module makes it possible to use
36+
`PgBouncer <https://pgbouncer.github.io/>`__ as a connection pooler for
37+
odoo.
38+
39+
Why isn't odoo's connection pooling good enough?
40+
------------------------------------------------
41+
42+
Odoo's builtin connection pooling works at process level: each Odoo
43+
process has its own
44+
`ConnectionPool <https://github.com/odoo/odoo/blob/12.0/odoo/sql_db.py#L525>`__,
45+
limited to ``db_maxconn``.
46+
47+
It does the job of re-using open connections available in the pool. But
48+
it never closes these connections, `unless reaching
49+
db_maxconn <https://github.com/odoo/odoo/blob/12.0/odoo/sql_db.py#L593>`__.
50+
51+
In practice, we observe that each odoo worker will end up with up to 3
52+
open connection in its pool. With 10 http workers, that's up to 30
53+
connection continuously open just for one single instance.
54+
55+
Here comes PgBouncer
56+
--------------------
57+
58+
PgBouncer will help to limit this number of open connections, by sharing
59+
a pool of connections at the instance level, between all workers. Odoo
60+
workers will still have up to 3 open connections, but these will be
61+
connections to PgBouncer, that on its side will close unnecessary
62+
connections to pg.
63+
64+
This has proven to help performances on Odoo deployments with multiple
65+
instances.
66+
67+
It allows you to define how resources should be shared, according to
68+
your priorities, e.g. :
69+
70+
- key odoo instance on host A can open up to 30 connections
71+
- while odoo instance on host B, dedicated to reports, can open up to 10
72+
connections only
73+
74+
And most importantly, it helps you to ensure that ``max_connections``
75+
will never be reached on pg server side.
76+
77+
Why is this module needed?
78+
--------------------------
79+
80+
When configuring PgBouncer, you can choose between 2 transaction pooling
81+
modes:
82+
83+
- pool_mode = session
84+
- pool_mode = transaction
85+
86+
If we choose pool_mode = session, then one server connection will be
87+
tied to a given odoo process until its death, which is exactly what
88+
we're trying to change. Thus, to release the server connection once the
89+
transaction is complete, we use pool_mode = transaction.
90+
91+
This works fine, except for Odoo's longpolling features that relies on
92+
the
93+
`LISTEN/NOTIFY <https://www.postgresql.org/docs/9.6/static/sql-notify.html>`__
94+
mechanism from pg, which is `not
95+
compatible <https://wiki.postgresql.org/wiki/PgBouncer>`__ with that
96+
mode.
97+
98+
To be more precise, NOTIFY statements are properly transfered by
99+
PgBouncer in that mode; only the LISTEN statement isn't (because it
100+
needs to keep the server connection open).
101+
102+
So for the unique "listening" connection per instance that requires this
103+
statement
104+
(`here <https://github.com/odoo/odoo/blob/12.0/addons/bus/models/bus.py#L166>`__),
105+
we need odoo to connect directly to the pg server, bypassing PgBouncer.
106+
107+
That's what this module implements, by overriding the relevant method of
108+
the
109+
`Dispatcher <https://github.com/odoo/odoo/blob/12.0/addons/bus/models/bus.py#L105>`__.
110+
111+
**Table of contents**
112+
113+
.. contents::
114+
:local:
115+
116+
Installation
117+
============
118+
119+
You don't need to install this module in the database(s) to enable it.
120+
121+
But you need to load it server-wide:
122+
123+
- By starting Odoo with ``--load=web,bus_alt_connection``
124+
- Or by updating its configuration file:
125+
126+
.. code:: ini
127+
128+
[options]
129+
(...)
130+
server_wide_modules = web,bus_alt_connection
131+
132+
Configuration
133+
=============
134+
135+
You need to define how to connect directly to the database:
136+
137+
- Either by defining environment variables:
138+
139+
- ``IMDISPATCHER_DB_HOST=db-01``
140+
- ``IMDISPATCHER_DB_PORT=5432``
141+
142+
- Or in Odoo's configuration file:
143+
144+
.. code:: ini
145+
146+
[options]
147+
(...)
148+
imdispatcher_db_host = db-01
149+
imdispatcher_db_port = 5432
150+
151+
Bug Tracker
152+
===========
153+
154+
Bugs are tracked on `GitHub Issues <https://github.com/OCA/server-tools/issues>`_.
155+
In case of trouble, please check there if your issue has already been reported.
156+
If you spotted it first, help us to smash it by providing a detailed and welcomed
157+
`feedback <https://github.com/OCA/server-tools/issues/new?body=module:%20bus_alt_connection%0Aversion:%2019.0%0A%0A**Steps%20to%20reproduce**%0A-%20...%0A%0A**Current%20behavior**%0A%0A**Expected%20behavior**>`_.
158+
159+
Do not contact contributors directly about support or help with technical issues.
160+
161+
Credits
162+
=======
163+
164+
Authors
165+
-------
166+
167+
* Trobz
168+
169+
Contributors
170+
------------
171+
172+
- Nils Hamerlinck <nils@trobz.com>
173+
174+
Maintainers
175+
-----------
176+
177+
This module is maintained by the OCA.
178+
179+
.. image:: https://odoo-community.org/logo.png
180+
:alt: Odoo Community Association
181+
:target: https://odoo-community.org
182+
183+
OCA, or the Odoo Community Association, is a nonprofit organization whose
184+
mission is to support the collaborative development of Odoo features and
185+
promote its widespread use.
186+
187+
This module is part of the `OCA/server-tools <https://github.com/OCA/server-tools/tree/19.0/bus_alt_connection>`_ project on GitHub.
188+
189+
You are welcome to contribute. To learn how please visit https://odoo-community.org/page/Contribute.

‎bus_alt_connection/__init__.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# Copyright 2019 Trobz <https://trobz.com>
2+
# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl).
3+
from . import models

‎bus_alt_connection/__manifest__.py‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
# Copyright 2019 Trobz <https://trobz.com>
2+
# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl).
3+
4+
{
5+
"name": "Bus Alt Connection",
6+
"summary": "Needed when using PgBouncer as a connection pooler",
7+
"version": "19.0.1.0.0",
8+
"author": "Trobz,Odoo Community Association (OCA)",
9+
"website": "https://github.com/OCA/server-tools",
10+
"category": "Extra Tools",
11+
"license": "AGPL-3",
12+
"depends": ["bus"],
13+
"installable": True,
14+
"auto_install": False,
15+
}

‎bus_alt_connection/i18n/bus_alt_connection.pot‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
# Translation of Odoo Server.
2+
# This file contains the translation of the following modules:
3+
#
4+
msgid ""
5+
msgstr ""
6+
"Project-Id-Version: Odoo Server 14.0\n"
7+
"Report-Msgid-Bugs-To: \n"
8+
"Last-Translator: \n"
9+
"Language-Team: \n"
10+
"MIME-Version: 1.0\n"
11+
"Content-Type: text/plain; charset=UTF-8\n"
12+
"Content-Transfer-Encoding: \n"
13+
"Plural-Forms: \n"

‎bus_alt_connection/models/__init__.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# Copyright 2019 Trobz <https://trobz.com>
2+
# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl).
3+
from . import bus

‎bus_alt_connection/models/bus.py‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
# Copyright 2019 Trobz <https://trobz.com>
2+
# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl).
3+
4+
import json
5+
import logging
6+
import os
7+
import selectors
8+
9+
import psycopg2
10+
11+
import odoo
12+
from odoo.tools import config
13+
14+
import odoo.addons.bus.models.bus
15+
from odoo.addons.bus.models.bus import TIMEOUT, hashable, stop_event
16+
17+
_logger = logging.getLogger(__name__)
18+
19+
20+
def _connection_info_for(db_name):
21+
db_or_uri, connection_info = odoo.sql_db.connection_info_for(db_name)
22+
23+
for p in ("host", "port"):
24+
cfg = os.environ.get(f"ODOO_IMDISPATCHER_DB_{p.upper()}") or config.get(
25+
"imdispatcher_db_" + p
26+
)
27+
if cfg:
28+
connection_info[p] = cfg
29+
return connection_info
30+
31+
32+
class ImDispatch(odoo.addons.bus.models.bus.ImDispatch):
33+
def loop(self):
34+
"""Dispatch postgres notifications to the relevant
35+
polling threads/greenlets"""
36+
connection_info = _connection_info_for("postgres")
37+
_logger.info(
38+
"Bus.loop listen imbus on db postgres (via %(host)s:%(port)s)",
39+
connection_info,
40+
)
41+
conn = psycopg2.connect(**connection_info)
42+
with conn.cursor() as cr, selectors.DefaultSelector() as sel:
43+
cr.execute("listen imbus")
44+
conn.commit()
45+
sel.register(conn, selectors.EVENT_READ)
46+
while not stop_event.is_set():
47+
if sel.select(TIMEOUT):
48+
conn.poll()
49+
channels = []
50+
while conn.notifies:
51+
channels.extend(json.loads(conn.notifies.pop().payload))
52+
# relay notifications to websockets that have
53+
# subscribed to the corresponding channels.
54+
websockets = set()
55+
for channel in channels:
56+
websockets.update(
57+
self._channels_to_ws.get(hashable(channel), [])
58+
)
59+
for websocket in websockets:
60+
websocket.trigger_notification_dispatching()
61+
62+
63+
odoo.addons.bus.models.bus.ImDispatch = ImDispatch
64+
dispatch = ImDispatch()
65+
odoo.addons.bus.models.bus.dispatch = dispatch
66+
odoo.addons.bus.models.ir_websocket.dispatch = dispatch
67+
odoo.addons.bus.websocket.dispatch = dispatch

‎bus_alt_connection/pyproject.toml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
[build-system]
2+
requires = ["whool"]
3+
build-backend = "whool.buildapi"

‎bus_alt_connection/readme/CONFIGURE.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
You need to define how to connect directly to the database:
2+
3+
- Either by defining environment variables:
4+
5+
> - `IMDISPATCHER_DB_HOST=db-01`
6+
> - `IMDISPATCHER_DB_PORT=5432`
7+
8+
- Or in Odoo's configuration file:
9+
10+
``` ini
11+
[options]
12+
(...)
13+
imdispatcher_db_host = db-01
14+
imdispatcher_db_port = 5432
15+
```

‎bus_alt_connection/readme/CONTRIBUTORS.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
- Nils Hamerlinck \<<nils@trobz.com>\>

‎bus_alt_connection/readme/DESCRIPTION.md‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
This module makes it possible to use
2+
[PgBouncer](https://pgbouncer.github.io/) as a connection pooler for
3+
odoo.
4+
5+
## Why isn't odoo's connection pooling good enough?
6+
7+
Odoo's builtin connection pooling works at process level: each Odoo
8+
process has its own
9+
[ConnectionPool](https://github.com/odoo/odoo/blob/12.0/odoo/sql_db.py#L525),
10+
limited to `db_maxconn`.
11+
12+
It does the job of re-using open connections available in the pool. But
13+
it never closes these connections, [unless reaching
14+
db_maxconn](https://github.com/odoo/odoo/blob/12.0/odoo/sql_db.py#L593).
15+
16+
In practice, we observe that each odoo worker will end up with up to 3
17+
open connection in its pool. With 10 http workers, that's up to 30
18+
connection continuously open just for one single instance.
19+
20+
## Here comes PgBouncer
21+
22+
PgBouncer will help to limit this number of open connections, by sharing
23+
a pool of connections at the instance level, between all workers. Odoo
24+
workers will still have up to 3 open connections, but these will be
25+
connections to PgBouncer, that on its side will close unnecessary
26+
connections to pg.
27+
28+
This has proven to help performances on Odoo deployments with multiple
29+
instances.
30+
31+
It allows you to define how resources should be shared, according to
32+
your priorities, e.g. :
33+
34+
- key odoo instance on host A can open up to 30 connections
35+
- while odoo instance on host B, dedicated to reports, can open up to 10
36+
connections only
37+
38+
And most importantly, it helps you to ensure that `max_connections` will
39+
never be reached on pg server side.
40+
41+
## Why is this module needed?
42+
43+
When configuring PgBouncer, you can choose between 2 transaction pooling
44+
modes:
45+
46+
- pool_mode = session
47+
- pool_mode = transaction
48+
49+
If we choose pool_mode = session, then one server connection will be
50+
tied to a given odoo process until its death, which is exactly what
51+
we're trying to change. Thus, to release the server connection once the
52+
transaction is complete, we use pool_mode = transaction.
53+
54+
This works fine, except for Odoo's longpolling features that relies on
55+
the
56+
[LISTEN/NOTIFY](https://www.postgresql.org/docs/9.6/static/sql-notify.html)
57+
mechanism from pg, which is [not
58+
compatible](https://wiki.postgresql.org/wiki/PgBouncer) with that mode.
59+
60+
To be more precise, NOTIFY statements are properly transfered by
61+
PgBouncer in that mode; only the LISTEN statement isn't (because it
62+
needs to keep the server connection open).
63+
64+
So for the unique "listening" connection per instance that requires this
65+
statement
66+
([here](https://github.com/odoo/odoo/blob/12.0/addons/bus/models/bus.py#L166)),
67+
we need odoo to connect directly to the pg server, bypassing PgBouncer.
68+
69+
That's what this module implements, by overriding the relevant method of
70+
the
71+
[Dispatcher](https://github.com/odoo/odoo/blob/12.0/addons/bus/models/bus.py#L105).

0 commit comments

Comments
 (0)