Etherpad-lite: Gracefully handle database connection unavailable

Created on 12 Nov 2020  ·  5Comments  ·  Source: ether/etherpad-lite

Describe the bug
When the database connection to the backend storage is unavailable (tested with postgres), the server keeps on crashing instead of gracefully reporting database connection issues.

To Reproduce
Steps to reproduce the behavior:

  1. Stop your database server (postgres)
  2. Start etherpad
  3. Visit some page that needs the database connection (any pad should do, in my case, also triggered when viewing /admin/plugins
  4. See error

Expected behavior
Graceful fallback to display some frontend error message, but keep running despite this lack of database connectivity.

Actual behaviour

[2020-11-12 22:13:05.858] [ERROR] console - error: column "value" is of type jsonb but expression is of type text
    at Parser.parseErrorMessage (/home/etherpad/etherpad-lite/src/node_modules/pg-protocol/dist/parser.js:278:15)
    at Parser.handlePacket (/home/etherpad/etherpad-lite/src/node_modules/pg-protocol/dist/parser.js:126:29)
    at Parser.parse (/home/etherpad/etherpad-lite/src/node_modules/pg-protocol/dist/parser.js:39:38)
    at Socket.stream.on (/home/etherpad/etherpad-lite/src/node_modules/pg-protocol/dist/index.js:8:42)
    at Socket.emit (events.js:198:13)
    at Socket.EventEmitter.emit (domain.js:448:20)
    at addChunk (_stream_readable.js:288:12)
    at readableAddChunk (_stream_readable.js:269:11)
    at Socket.Readable.push (_stream_readable.js:224:10)
    at TCP.onStreamRead [as onread] (internal/stream_base_commons.js:94:17)
[2020-11-12 22:13:05.858] [INFO] console - Stopping Etherpad...
[2020-11-12 22:13:05.876] [ERROR] console - TypeError: Cannot read property 'name' of null
    at Client._handleParseComplete (/home/etherpad/etherpad-lite/src/node_modules/pg/lib/client.js:358:26)
    at Connection.emit (events.js:198:13)
    at Connection.EventEmitter.emit (domain.js:448:20)
    at parse (/home/etherpad/etherpad-lite/src/node_modules/pg/lib/connection.js:109:12)
    at Parser.parse (/home/etherpad/etherpad-lite/src/node_modules/pg-protocol/dist/parser.js:40:17)
    at Socket.stream.on (/home/etherpad/etherpad-lite/src/node_modules/pg-protocol/dist/index.js:8:42)
    at Socket.emit (events.js:198:13)
    at Socket.EventEmitter.emit (domain.js:448:20)
    at addChunk (_stream_readable.js:288:12)
    at readableAddChunk (_stream_readable.js:269:11)

Additional context

Etherpad version
Version number: 1.8.6
Latest available version: 1.8.6
Git sha: 6a8563e
picture BenBE

Most helpful comment

Oh check the reverted commits. There was a bad pg commit a while back that was related I think..

picture JohnMcLear on 12 Nov 2020
👍2

All 5 comments

+1 this.

https://github.com/ether/ueberDB for our database abstraction btw. This isn't just a PG issue, affects all DBs.

I'd accept a PR, I'm not sure where is best to fix it, I guess prolly have ueberdb emit when db is lost.

You would prolly never wanna bubble a msg up to the client tho so maybe just generic error to client, clear error in backend logs and cache to memory until database is restored.. The problem is that cache to memory until restored database prolly requires new logic for each database connector which may be a bigger job than we have time for..

It's worth worth noting that Etherpad is designed to die on error, we try not to keep things alive but failing. It was an early design decision that paid off pretty well as it forced us to make Etherpad stable(ish).. Etherpad's startup scripts will try to restart Etherpad and re-establish db connectivity, I'm not opposed to how we do things but I feel like if the database is gone it should be very clear that Etherpad is failing because a database has gone away.

JohnMcLear picture JohnMcLear on 12 Nov 2020
👍1

I don't think this is a connectivity issue. column <name> is of type jsonb but expression is of type text seems to be a rather common thing that happens when the middleware sets up query parameters in unexpected ways. The column is created as JSONB, but when the query generator uses something like JSON.encode, the type is inferred to be a string and the parameter is bound as type TEXT or VARCHAR ("character varying" in error messages).
Apparently that can also be a problem when storing strings or arrays instead of objects containing them?

Random example found via Google, not related to this project: https://github.com/jeremydaly/data-api-client/issues/27#issuecomment-584701957

martok picture martok on 12 Nov 2020
👍1

Oh check the reverted commits. There was a bad pg commit a while back that was related I think..

JohnMcLear picture JohnMcLear on 12 Nov 2020
👍2

I just noticed -- looks like we got the CREATE TABLE with jsonb in it, the version in node_modules is just text already. FWIW we also have ueberdb_insert_or_update() with text argument, no overloaded one remaining...

Just to be sure before I convert it, text is what it's supposed to be (for now)?

martok picture martok on 13 Nov 2020
👍1

After conversion to text everything seems to work:

ALTER TABLE storage ALTER COLUMN value TYPE text USING value::text;
BenBE picture BenBE on 13 Nov 2020
👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

julpec picture julpec  ·  9Comments
ziyaointl picture ziyaointl  ·  9Comments
yurivict picture yurivict  ·  8Comments
muxator picture muxator  ·  7Comments
kernelfreak picture kernelfreak  ·  9Comments