This line is sending a websocket message for every column, based on equality
Some columns can be huge text dumps or JSON config so we should add some sensible filters filters here.
Note from Fran:
I can reproduce by inserting huge columns into my local db. First realtime takes a while to pickup the row, then the event is never sent through channels.
Here are some actual row from one of our customers - worth testing on:
| id | url | config | project | createdAt | updatedAt | error | extractedAt | data | owner |
| ------------------------------------ | ---------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------ | ------------------------ | ------------------------ | ----- | ------------------------ | --------------------------------------------------------------------------------------- | ------------------------------------ |
| 00cbfb63-58ea-49b5-83b9-9340e8f2ea03 | https://github.com/supabase/postgrest-rs | {"prs":"[data-tab-item=\"i2pull-requests-tab\"] > span:nth-child(3) ","fork":"li:nth-child(4) > a:nth-child(2) ","star":".social-count[href=\"stargazers\"]","watch":"li:nth-child(2) > a:nth-child(2) ","issues":"[data-tab-item=\"i1issues-tab\"] > span:nth-child(3) ","selectors":{"prs":"[data-tab-item=\"i2pull-requests-tab\"] > span:nth-child(3) ","fork":"li:nth-child(4) > a:nth-child(2) ","star":".social-count[href=\"stargazers\"]","watch":"li:nth-child(2) > a:nth-child(2) ","issues":"[data-tab-item=\"i1issues-tab\"] > span:nth-child(3) ","isSponsored":"[id=\"sponsor-button-repo\"] "},"isSponsored":"[id=\"sponsor-button-repo\"] "} | 9ef3b703-4d65-4670-8c03-5870e490cb75 | 2020-12-06T08:08:43.979Z | 2020-12-06T08:08:44.131Z | | 2021-02-10T03:32:29.564Z | {"prs":"1","fork":"1","star":"51","watch":"6","issues":"1","isSponsored":"Sponsor"} | 615f7380-58e7-4c81-82f9-3551e1e23c68 |
| 8cfe1c3c-8847-4eae-91b0-a4f64a62817f | https://github.com/supabase/realtime | {"prs":"[data-tab-item=\"i2pull-requests-tab\"] > span:nth-child(3) ","fork":"li:nth-child(4) > a:nth-child(2) ","star":".social-count[href=\"stargazers\"]","watch":"li:nth-child(2) > a:nth-child(2) ","issues":"[data-tab-item=\"i1issues-tab\"] > span:nth-child(3) ","selectors":{"prs":"[data-tab-item=\"i2pull-requests-tab\"] > span:nth-child(3) ","fork":"li:nth-child(4) > a:nth-child(2) ","star":".social-count[href=\"stargazers\"]","watch":"li:nth-child(2) > a:nth-child(2) ","issues":"[data-tab-item=\"i1issues-tab\"] > span:nth-child(3) ","isSponsored":"[id=\"sponsor-button-repo\"] "},"isSponsored":"[id=\"sponsor-button-repo\"] "} | 9ef3b703-4d65-4670-8c03-5870e490cb75 | 2020-12-06T08:08:43.976Z | 2020-12-06T08:08:44.122Z | | 2021-02-10T03:32:35.548Z | {"prs":"4","fork":"95","star":"2.2k","watch":"66","issues":"8","isSponsored":"Sponsor"} | 615f7380-58e7-4c81-82f9-3551e1e23c68 |
The name of the channels is value in ETS, it can be really huge, even GBs.
As for that bug, I think it's related to that checking:
https://github.com/supabase/realtime/blob/894f4bb8923017467c78803711d8adbef8c090fe/server/lib/realtime/subscribers_notification.ex#L140-L142
In current implementation can be checked keys only up to 100 symbols.
Nice thanks @abc3.
I should check my own code before creating issues 😅
I'll set up some tests to make this fail - @fracek confirmed it was an issue, so it's worth figuring out where the failure is occuring.
I've checked with key 2500 length, everything ok. I'm with pleasure will investigate bug if you show how to reproduce it. :blush:
I start with a fresh copy of realtime (commit 894f4bb89230
), get the deps for the server (mix deps.get
) and for the node js example (yarn install
). I start the db from the node-js example (docker-compose up db
).
I start the server:
PORT=4000 \
HOSTNAME=localhost \
DB_USER=postgres \
DB_HOST=localhost \
DB_PASSWORD=postgres \
DB_NAME=postgres \
DB_PORT=5432 \
DB_PORT=5432 \
SLOT_NAME=TEST_SLOT \
mix phx.server
and the node-js example (yarn run start
).
I insert some data with the following script:
import psycopg2
import time
def main():
conn = psycopg2.connect(
host="localhost",
database="postgres",
user="postgres",
password="postgres"
)
cur = conn.cursor()
name = 'name'
cur.execute("INSERT INTO users(name) VALUES(%s)", (name, ))
conn.commit()
main()
and I can see the data streamed to the node-js example. I change the string length to 4000 (name = 'name' * 1000
) and it works. I increase the size of the string incrementally (10 in 10) until name = 'name' * 1000000
(1_000_000
). I changed the script to first insert a big string and after that a small string, in this case node-js does not receive either of them.
import psycopg2
import time
def main():
conn = psycopg2.connect(
host="localhost",
database="postgres",
user="postgres",
password="postgres"
)
cur = conn.cursor()
name = 'name' * 1000000
cur.execute("INSERT INTO users(name) VALUES(%s)", (name, ))
conn.commit()
name = 'name'
cur.execute("INSERT INTO users(name) VALUES(%s)", (name, ))
conn.commit()
main()
I believe the problem is in the channels not being able to handle big payloads.
One last experiment is to check the replication lag. I use the following command:
select
slot_name, pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS replicationSlotLag, active
from pg_replication_slots ;
Normally the result is:
slot_name | replicationslotlag | active
-----------+--------------------+--------
test_slot | 56 bytes | t
(1 row)
I run my python script once and see it growing (every time I run the script it grows by about 40kB). @w3b6x9 showed me that on the customer instance this size is around 20MB. After a while the realtime instance gets killed by the os (with SIGKILL). I restart the instance and it slowly (very slowly) goes through the backlog but it gets killed regularly because the memory usage keeps increasing.
To recap there are two visible issues when inserting big rows:
My educated guess is that there are two main issues (maybe related, maybe not):
But name = 'name' * 1000000
is only 40MB :confused:
In my case it works, I'll dive deeper with @fracek example.
Btw, my test shows a very big latency of ~10sec. It's too much for deliverу a few tens MB.
export default async (req, res) => {
const values = Array(1000000).fill("name").join('')
const text = `INSERT INTO stress(value) VALUES('${values}') RETURNING id`
const q = await pool.query(text)
res.json(JSON.stringify(q.rows))
}
thanks for testing @abc3 - it could be due to this: https://github.com/supabase/realtime/pull/120
We were seeing this behaviour in Prod, so had to revert the PR. If it's still slow after merging this PR, then we definitely need to fix the latency
I've made a mistake in my calculation: not 40 but 4MB ))
testing insert through this INSERT INTO users(name) VALUES(repeat('name', 1000000))
I've found it!
Response time with debug level and without:
These rows execute few seconds:
https://github.com/supabase/realtime/blob/894f4bb8923017467c78803711d8adbef8c090fe/server/lib/realtime/replication.ex#L72-L73
I can reproduce the fix. Events are still not propagated through websockets but at least the instance doesn't become unresponsive. Good job!
The browser needs some time to decode and render long strings.
It will be clearer if make some changes in the example.
need to remove this:
https://github.com/supabase/realtime/blob/894f4bb8923017467c78803711d8adbef8c090fe/examples/next-js/pages/index.js#L28
replace that
https://github.com/supabase/realtime/blob/894f4bb8923017467c78803711d8adbef8c090fe/examples/next-js/pages/index.js#L44-L53
with
messageReceived(channel, msg) {
console.log('got message')
}
and read messages in the browser console
cur = conn.cursor()
name = 'name' * 1000000
cur.execute("INSERT INTO users(name) VALUES(%s)", (name, ))
conn.commit()
@fracek weird that this was giving you issues on your machine.
I went to check it out while I was debugging. I inserted 100 rows, with each row having value 'name' * 1000000
(4MB), in the same transaction.
As replication was in progress, I checked both the replication lag (stayed consistent at about 5.3MB):
and the flush lag:
_This is similar to what I saw while debugging the customer's db instance yesterday. Their replication lag would climb and then stay consistent at a lag (12MB, 16MB, 20MB, etc.) and have a flush lag anywhere between 3-5 minutes._
After waiting for quite a while I got the messages. Each one contained the 'name' * 1000000
(4MB) value. I also tracked how long replication took (~30 min.) compared to sending out 100 messages each carrying the 4MB value (~90 seconds).
I did find someone with the issue Disconnected from Phoenix channel when sending multiple large messages in a small time frame. One user mentioned:
- Phoenix I think uses a 60s or so timeout to receive a heartbeat or so message.
- If it takes longer than that to send a single message, then phoenix thinks the pipe is frozen or overwhelmed or something, so it kills it.
And another user, sasajuric
, offered the following solution that worked for him:
- compressing/decompressing the payload outside of the serializer
- serializing/deserializing messages using :erlang.term_to_binary and :erlang.binary_to_term
For now, just something to keep in mind but If we do discover that messages are not being sent due to high frequency and huge payload, then the first thing we can do is use the newer built-in serializer Phoenix.Socket.V2.JSONSerializer
(we're currently using V1
) before doing what sasajuric
suggested.
it could be due to this: #120
Yea, if realtime
server kept on restarting and picking up where it left off with the permanent replication slot then I can see it taking longer and longer (if ever in the case that it couldn't get back to a working state due to :undefined.handle_message/4
errors) for the customer to receive messages.
This might just be normal considering the customer performs huge dumps of updates at once (@kiwicopple I think you had mentioned this somewhere). #120 has been merged and will be deployed for the customer shortly so we'll just have to continue monitoring.
@abc3: btw, your Nextjs example with chart of routing time #118 is awesome and was super helpful when I was debugging. Thanks again!
Thanks! I'm glad that my contribution is helpful :blush:
Please, share your monitoring result if something goes wrong on prod again.
The old quick solution adds time too https://github.com/supabase/realtime/issues/8#issuecomment-564551365
https://github.com/supabase/realtime/blob/af6344c7746e8a8af6a11a9b498721c1f97e339b/server/lib/realtime_web/channels/realtime_channel.ex#L21-L24
So, the code above and the implementation Realtime.SubscribersNotification.notify_subscribers
will send several copies of the similar data.
For example, when inserted a row in the table users
the client with subscriptions to this.addChannel('realtime:*')
and this.addChannel('realtime:public:users')
will receive 4 messages.
If one message 4MB, the server will send 16MB
Most helpful comment
I've found it!
Response time with debug level and without:
These rows execute few seconds:
https://github.com/supabase/realtime/blob/894f4bb8923017467c78803711d8adbef8c090fe/server/lib/realtime/replication.ex#L72-L73