Etherpad-lite: Postgres backend - error: column "value" is of type jsonb but expression is of type text

Created on 8 Sep 2020  ·  29Comments  ·  Source: ether/etherpad-lite

Describe the bug
Schema mismatch when doing any kind of inserts.

error: column "value" is of type jsonb but expression is of type text

To Reproduce
Use latest source and postgres. Tried with v12 and v13.

Expected behavior
To do the insert without error.

Screenshots
etherpad_1 | [2020-09-08 20:37:50.541] [ERROR] console - error: column "value" is of type jsonb but expression is of type text
etherpad_1 | at Parser.parseErrorMessage (/opt/etherpad-lite/src/node_modules/pg-protocol/dist/parser.js:278:15)
etherpad_1 | at Parser.handlePacket (/opt/etherpad-lite/src/node_modules/pg-protocol/dist/parser.js:126:29)
etherpad_1 | at Parser.parse (/opt/etherpad-lite/src/node_modules/pg-protocol/dist/parser.js:39:38)
etherpad_1 | at Socket.stream.on (/opt/etherpad-lite/src/node_modules/pg-protocol/dist/index.js:8:42)
etherpad_1 | at Socket.emit (events.js:198:13)
etherpad_1 | at Socket.EventEmitter.emit (domain.js:448:20)
etherpad_1 | at addChunk (_stream_readable.js:288:12)
etherpad_1 | at readableAddChunk (_stream_readable.js:269:11)
etherpad_1 | at Socket.Readable.push (_stream_readable.js:224:10)
etherpad_1 | at TCP.onStreamRead [as onread] (internal/stream_base_commons.js:94:17)

Desktop (please complete the following information):
NA

Smartphone (please complete the following information):
NA

Additional context
This was tested out with docker containers.

Serious Bug

Most helpful comment

Thanks @JohnMcLear , running the following made it work again:

alter table "store" alter column "value" type text;

All 29 comments

This is mostly an issue where I used latest source when I should've used a release.

I've tried this with 1.8.4 and can confirm I do not get this error. I have updated the issue accordingly.

Except 1.8.5 does include this bug, so whoops.

Fixed in c8361a211a1e6bfbbc006933f48d48ca51857c71

Fixed in c8361a2

I still get that.

image

Did you git pull then bin/run.sh?

You will need to update ueber and bin/run.sh will do that. Don't use fastrun

Yeah, I always come up with bin/run.sh.
It is updated to "ueberdb2": "^0.5.2"

Screenshot_20200910_103721

Tests are passing fine.

Can you try a new pg dB plz? I want to see if previous commit broke your database.

Can you try a new pg dB plz? I want to see if previous commit broke your database.

sure, let me try.

Can you try a new pg dB plz? I want to see if previous commit broke your database.

Actually, due to my dev environment, I am getting a new pg db on every build.
image
image

Then Peep Travis config to see how we create the database for the ueberdb tests. You will want the ueber dB repo not here

Does it seem we should use the below repo?
https://github.com/ether/ueberDB#dirtyref

dirtyref is merged into master. Use master

Hi, I have this issue on 1.8.5 as well. Can you re-open this issue until a solution is found? (so it's easier to find it!)
Thanks!

cc @Gared

I still have this issue even change to repo https://github.com/ether/ueberDB (master)

image

Same here:

  • Pads are not loading (either eternal "Loading…" or 502)
  • Server fails every hour with this exception:
[2020-09-11 11:36:24.449] [ERROR] console - error: column "value" is of type jsonb but expression is of type text
    at Parser.parseErrorMessage (/usr/local/var/etherpad/repo/src/node_modules/pg-protocol/dist/parser.js:278:15)
    at Parser.handlePacket (/usr/local/var/etherpad/repo/src/node_modules/pg-protocol/dist/parser.js:126:29)
    at Parser.parse (/usr/local/var/etherpad/repo/src/node_modules/pg-protocol/dist/parser.js:39:38)
    at Socket.<anonymous> (/usr/local/var/etherpad/repo/src/node_modules/pg-protocol/dist/index.js:8:42)
    at Socket.emit (events.js:315:20)
    at Socket.EventEmitter.emit (domain.js:483:12)
    at addChunk (_stream_readable.js:295:12)
    at readableAddChunk (_stream_readable.js:271:9)
    at Socket.Readable.push (_stream_readable.js:212:10)
    at TCP.onStreamRead (internal/stream_base_commons.js:186:23)
[2020-09-11 11:36:24.463] [ERROR] console - TypeError: Cannot read property 'name' of null
    at Client._handleParseComplete (/usr/local/var/etherpad/repo/src/node_modules/pg/lib/client.js:358:26)
    at Connection.emit (events.js:315:20)
    at Connection.EventEmitter.emit (domain.js:483:12)
    at /usr/local/var/etherpad/repo/src/node_modules/pg/lib/connection.js:109:12
    at Parser.parse (/usr/local/var/etherpad/repo/src/node_modules/pg-protocol/dist/parser.js:40:17)
    at Socket.<anonymous> (/usr/local/var/etherpad/repo/src/node_modules/pg-protocol/dist/index.js:8:42)
    at Socket.emit (events.js:315:20)
    at Socket.EventEmitter.emit (domain.js:483:12)
    at addChunk (_stream_readable.js:295:12)
    at readableAddChunk (_stream_readable.js:271:9)
  • OS: Ubuntu 18.04
  • settings.json:
{
  "dbSettings" : {
                   "user"    : "etherpad",
                   "host"    : "localhost",
                   "port"    : 5432,
                   "password": "xxx",
                   "database": "etherpad",
                 },
  "padOptions": {
    "noColors":         false,
    "showControls":     true,
    "showChat":         true,
    "showLineNumbers":  true,
    "useMonospaceFont": false,
    "userName":         false,
    "userColor":        false,
    "rtl":              false,
    "alwaysShowChat":   false,
    "chatAndUsers":     false,
    "lang":             "de-de"
  },
  "suppressErrorsInPadText": false,
  "requireSession": false,
  "editOnly": true,
  "sessionNoPassword": false,
  "minify": true,
  "maxAge": 21600,
  "abiword": null,
  "soffice": null,
  "tidyHtml": null,
  "allowUnknownFileEnds": true,
  "requireAuthentication": false,
  "requireAuthorization": false,
  "trustProxy": true,
  "disableIPlogging": true,
  "automaticReconnectionTimeout": 5,
  "users": {
    "admin": {
      "hash": "xxx",
      "is_admin": true
    }
  },
  "socketTransportProtocols" : ["xhr-polling", "jsonp-polling", "htmlfile"],
  "loadTest": false,
  "exposeVersion": false,
  "loglevel": "WARN",
  "logconfig" :
    { "appenders": [
        { "type": "console" }
      ]
    }
}
  • DB: postgresql 10+190ubuntu0.1
  • DB schema:
SET statement_timeout = 0;
SET lock_timeout = 0;
SET idle_in_transaction_session_timeout = 0;
SET client_encoding = 'UTF8';
SET standard_conforming_strings = on;
SELECT pg_catalog.set_config('search_path', '', false);
SET check_function_bodies = false;
SET xmloption = content;
SET client_min_messages = warning;
SET row_security = off;
CREATE EXTENSION IF NOT EXISTS plpgsql WITH SCHEMA pg_catalog;
COMMENT ON EXTENSION plpgsql IS 'PL/pgSQL procedural language';
CREATE FUNCTION public.ueberdb_insert_or_update(character varying, text) RETURNS void
    LANGUAGE plpgsql
    AS $_$ BEGIN   IF EXISTS( SELECT * FROM store WHERE key = $1 ) THEN     UPDATE store SET value = $2 WHERE key = $1;   ELSE     INSERT INTO store(key,value) VALUES( $1, $2 );   END IF;   RETURN; END; $_$;
ALTER FUNCTION public.ueberdb_insert_or_update(character varying, text) OWNER TO etherpad;
CREATE FUNCTION public.ueberdb_insert_or_update_jsonb(character varying, jsonb) RETURNS void
    LANGUAGE plpgsql
    AS $_$ BEGIN   IF EXISTS( SELECT * FROM store WHERE key = $1 ) THEN     UPDATE store SET value = $2 WHERE key = $1;   ELSE     INSERT INTO store(key,value) VALUES( $1, $2 );   END IF;   RETURN; END; $_$;
ALTER FUNCTION public.ueberdb_insert_or_update_jsonb(character varying, jsonb) OWNER TO etherpad;
SET default_tablespace = '';
SET default_with_oids = false;
CREATE TABLE public.store (
    key character varying(100) NOT NULL,
    value jsonb NOT NULL
);
ALTER TABLE public.store OWNER TO etherpad;
ALTER TABLE ONLY public.store
    ADD CONSTRAINT store_pkey PRIMARY KEY (key);

I took the rollback road. I tried to rollback just etherpad to 1.8.4 but it didn't work:

[2020-09-11 20:55:44.494] [ERROR] console - JSON-PROBLEM:[object Object]
[2020-09-11 20:55:44.495] [ERROR] console - (node:1) UnhandledPromiseRejectionWarning: SyntaxError: Unexpected token o in JSON at position 1
    at JSON.parse (<anonymous>)
    at /opt/etherpad-lite/src/node_modules/ueberdb2/CacheAndBufferLayer.js:169:24
    at Query.<anonymous> (/opt/etherpad-lite/src/node_modules/ueberdb2/postgres_common.js:99:5)
    at Query.handleReadyForQuery (/opt/etherpad-lite/src/node_modules/pg/lib/query.js:128:10)
    at Connection.<anonymous> (/opt/etherpad-lite/src/node_modules/pg/lib/client.js:247:19)
    at Connection.emit (events.js:198:13)
    at Connection.EventEmitter.emit (domain.js:448:20)
    at Socket.<anonymous> (/opt/etherpad-lite/src/node_modules/pg/lib/connection.js:134:12)
    at Socket.emit (events.js:198:13)
    at Socket.EventEmitter.emit (domain.js:448:20)
[2020-09-11 20:55:44.496] [ERROR] console - (node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
[2020-09-11 20:55:44.496] [ERROR] console - (node:1) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

So I restored a pgsql dump as well and now 1.8.4 works again (which is not really a surprise since it had been working finely).

reason: from 1.8.4 to 1.8.5 the column value is changed from type text to jsonb.
Should be a major release...

Is there a migration script which can be applied?

I'm just gonna revert these commits. This was all poorly handled IMHO.

The change log says "IMPORTANT: This fixes a severe problem with postgresql in 1.8.5" for 1.8.6, but I've upgraded and still get the exceptions mentioned above. I now noticed these exceptions as well, which are output immediately after service restart:

[2020-09-19 17:47:45.482] [ERROR] console - JSON-PROBLEM:[object Object]                                    
[2020-09-19 17:47:45.496] [ERROR] console - (node:30613) UnhandledPromiseRejectionWarning: SyntaxError: Unexpected token o in JSON at position 1
    at JSON.parse (<anonymous>)                                                                         
    at /usr/local/var/etherpad/repo/src/node_modules/ueberdb2/lib/CacheAndBufferLayer.js:169:24
    at Query.<anonymous> (/usr/local/var/etherpad/repo/src/node_modules/ueberdb2/databases/postgres_common.js:99:5)
    at Query.handleReadyForQuery (/usr/local/var/etherpad/repo/src/node_modules/pg/lib/query.js:122:12)
    at Client._handleReadyForQuery (/usr/local/var/etherpad/repo/src/node_modules/pg/lib/client.js:281:19)
    at Connection.emit (events.js:315:20)               
    at Connection.EventEmitter.emit (domain.js:483:12)          
    at /usr/local/var/etherpad/repo/src/node_modules/pg/lib/connection.js:109:12          
    at Parser.parse (/usr/local/var/etherpad/repo/src/node_modules/pg-protocol/dist/parser.js:40:17)      
    at Socket.<anonymous> (/usr/local/var/etherpad/repo/src/node_modules/pg-protocol/dist/index.js:8:42)
[2020-09-19 17:47:45.497] [ERROR] console - (node:30613) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)    
[2020-09-19 17:47:45.497] [ERROR] console - (node:30613) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

@c7hm4r did your database change or something? If you start a new tarble/database are things okay?

@JohnMcLear When restart Etherpad 1.8.6 with an empty database, a new schema is created and it seems to work without exceptions.

When I compare the new schema with the old one, I get following:

diff --git a/etherpad_schema.sql b/etherpad_schema_new.sql
index 9baf5f4..ad3c08b 100644
--- a/etherpad_schema.sql
+++ b/etherpad_schema_new.sql
@@ -41,17 +41,6 @@ CREATE FUNCTION public.ueberdb_insert_or_update(character varying, text) RETURNS

 ALTER FUNCTION public.ueberdb_insert_or_update(character varying, text) OWNER TO etherpad;

---
--- Name: ueberdb_insert_or_update_jsonb(character varying, jsonb); Type: FUNCTION; Schema: public; Owner: etherpad
---
-
-CREATE FUNCTION public.ueberdb_insert_or_update_jsonb(character varying, jsonb) RETURNS void
-    LANGUAGE plpgsql
-    AS $_$ BEGIN   IF EXISTS( SELECT * FROM store WHERE key = $1 ) THEN     UPDATE store SET value = $2 WHERE key = $1;   ELSE     INSERT INTO store(key,value) VALUES( $1, $2 );   END IF;   RETURN; END; $_$;
-
-
-ALTER FUNCTION public.ueberdb_insert_or_update_jsonb(character varying, jsonb) OWNER TO etherpad;
-
 SET default_tablespace = '';

 SET default_with_oids = false;
@@ -62,7 +51,7 @@ SET default_with_oids = false;

 CREATE TABLE public.store (
     key character varying(100) NOT NULL,
-    value jsonb NOT NULL
+    value text NOT NULL
 );

So the question is: When the type of the column has changed and how to migrate/revert that?

I configured automatic updates for my Etherpad instance from the master branch.

@c7hm4r @qsuscs was the author of the offending commits. I'm hoping he takes some responsibility here and helps out :)

https://github.com/ether/ueberDB/commit/b9e786e2c8753f4cbbe09d3695a92a1a03904ead#diff-f1b8245dd259fa912c33cb2db527f609R31 is the alter statement..

  var updateValueColumnType = "ALTER TABLE store " +
    "ALTER COLUMN value " +
    "TYPE jsonb " +
    "USING value::jsonb";

One would assume changing jsonb to text would do it?

Thanks @JohnMcLear , running the following made it work again:

alter table "store" alter column "value" type text;

@c7hm4r no worries, hopefully the handful of people this affects will find this thread and do that command., Thanks for writing and testing! :) I'd be willing to take a PR to fix if you or the original author wanted. I have other tasks at the moment that are taking priority.

Was this page helpful?
0 / 5 - 0 ratings