Describe the bug
Schema mismatch when doing any kind of inserts.
error: column "value" is of type jsonb but expression is of type text
To Reproduce
Use latest source and postgres. Tried with v12 and v13.
Expected behavior
To do the insert without error.
Screenshots
etherpad_1 | [2020-09-08 20:37:50.541] [ERROR] console - error: column "value" is of type jsonb but expression is of type text
etherpad_1 | at Parser.parseErrorMessage (/opt/etherpad-lite/src/node_modules/pg-protocol/dist/parser.js:278:15)
etherpad_1 | at Parser.handlePacket (/opt/etherpad-lite/src/node_modules/pg-protocol/dist/parser.js:126:29)
etherpad_1 | at Parser.parse (/opt/etherpad-lite/src/node_modules/pg-protocol/dist/parser.js:39:38)
etherpad_1 | at Socket.stream.on (/opt/etherpad-lite/src/node_modules/pg-protocol/dist/index.js:8:42)
etherpad_1 | at Socket.emit (events.js:198:13)
etherpad_1 | at Socket.EventEmitter.emit (domain.js:448:20)
etherpad_1 | at addChunk (_stream_readable.js:288:12)
etherpad_1 | at readableAddChunk (_stream_readable.js:269:11)
etherpad_1 | at Socket.Readable.push (_stream_readable.js:224:10)
etherpad_1 | at TCP.onStreamRead [as onread] (internal/stream_base_commons.js:94:17)
Desktop (please complete the following information):
NA
Smartphone (please complete the following information):
NA
Additional context
This was tested out with docker containers.
This is mostly an issue where I used latest source when I should've used a release.
I've tried this with 1.8.4 and can confirm I do not get this error. I have updated the issue accordingly.
Except 1.8.5 does include this bug, so whoops.
Fixed in c8361a211a1e6bfbbc006933f48d48ca51857c71
Fixed in c8361a2
I still get that.
Did you git pull then bin/run.sh?
You will need to update ueber and bin/run.sh will do that. Don't use fastrun
Yeah, I always come up with bin/run.sh
.
It is updated to "ueberdb2": "^0.5.2"
Tests are passing fine.
Can you try a new pg dB plz? I want to see if previous commit broke your database.
Can you try a new pg dB plz? I want to see if previous commit broke your database.
sure, let me try.
Can you try a new pg dB plz? I want to see if previous commit broke your database.
Actually, due to my dev environment, I am getting a new pg db on every build.
Then Peep Travis config to see how we create the database for the ueberdb tests. You will want the ueber dB repo not here
Does it seem we should use the below repo?
https://github.com/ether/ueberDB#dirtyref
master
Hi, I have this issue on 1.8.5 as well. Can you re-open this issue until a solution is found? (so it's easier to find it!)
Thanks!
cc @Gared
I still have this issue even change to repo https://github.com/ether/ueberDB (master)
Same here:
[2020-09-11 11:36:24.449] [ERROR] console - error: column "value" is of type jsonb but expression is of type text
at Parser.parseErrorMessage (/usr/local/var/etherpad/repo/src/node_modules/pg-protocol/dist/parser.js:278:15)
at Parser.handlePacket (/usr/local/var/etherpad/repo/src/node_modules/pg-protocol/dist/parser.js:126:29)
at Parser.parse (/usr/local/var/etherpad/repo/src/node_modules/pg-protocol/dist/parser.js:39:38)
at Socket.<anonymous> (/usr/local/var/etherpad/repo/src/node_modules/pg-protocol/dist/index.js:8:42)
at Socket.emit (events.js:315:20)
at Socket.EventEmitter.emit (domain.js:483:12)
at addChunk (_stream_readable.js:295:12)
at readableAddChunk (_stream_readable.js:271:9)
at Socket.Readable.push (_stream_readable.js:212:10)
at TCP.onStreamRead (internal/stream_base_commons.js:186:23)
[2020-09-11 11:36:24.463] [ERROR] console - TypeError: Cannot read property 'name' of null
at Client._handleParseComplete (/usr/local/var/etherpad/repo/src/node_modules/pg/lib/client.js:358:26)
at Connection.emit (events.js:315:20)
at Connection.EventEmitter.emit (domain.js:483:12)
at /usr/local/var/etherpad/repo/src/node_modules/pg/lib/connection.js:109:12
at Parser.parse (/usr/local/var/etherpad/repo/src/node_modules/pg-protocol/dist/parser.js:40:17)
at Socket.<anonymous> (/usr/local/var/etherpad/repo/src/node_modules/pg-protocol/dist/index.js:8:42)
at Socket.emit (events.js:315:20)
at Socket.EventEmitter.emit (domain.js:483:12)
at addChunk (_stream_readable.js:295:12)
at readableAddChunk (_stream_readable.js:271:9)
settings.json
:{
"dbSettings" : {
"user" : "etherpad",
"host" : "localhost",
"port" : 5432,
"password": "xxx",
"database": "etherpad",
},
"padOptions": {
"noColors": false,
"showControls": true,
"showChat": true,
"showLineNumbers": true,
"useMonospaceFont": false,
"userName": false,
"userColor": false,
"rtl": false,
"alwaysShowChat": false,
"chatAndUsers": false,
"lang": "de-de"
},
"suppressErrorsInPadText": false,
"requireSession": false,
"editOnly": true,
"sessionNoPassword": false,
"minify": true,
"maxAge": 21600,
"abiword": null,
"soffice": null,
"tidyHtml": null,
"allowUnknownFileEnds": true,
"requireAuthentication": false,
"requireAuthorization": false,
"trustProxy": true,
"disableIPlogging": true,
"automaticReconnectionTimeout": 5,
"users": {
"admin": {
"hash": "xxx",
"is_admin": true
}
},
"socketTransportProtocols" : ["xhr-polling", "jsonp-polling", "htmlfile"],
"loadTest": false,
"exposeVersion": false,
"loglevel": "WARN",
"logconfig" :
{ "appenders": [
{ "type": "console" }
]
}
}
SET statement_timeout = 0;
SET lock_timeout = 0;
SET idle_in_transaction_session_timeout = 0;
SET client_encoding = 'UTF8';
SET standard_conforming_strings = on;
SELECT pg_catalog.set_config('search_path', '', false);
SET check_function_bodies = false;
SET xmloption = content;
SET client_min_messages = warning;
SET row_security = off;
CREATE EXTENSION IF NOT EXISTS plpgsql WITH SCHEMA pg_catalog;
COMMENT ON EXTENSION plpgsql IS 'PL/pgSQL procedural language';
CREATE FUNCTION public.ueberdb_insert_or_update(character varying, text) RETURNS void
LANGUAGE plpgsql
AS $_$ BEGIN IF EXISTS( SELECT * FROM store WHERE key = $1 ) THEN UPDATE store SET value = $2 WHERE key = $1; ELSE INSERT INTO store(key,value) VALUES( $1, $2 ); END IF; RETURN; END; $_$;
ALTER FUNCTION public.ueberdb_insert_or_update(character varying, text) OWNER TO etherpad;
CREATE FUNCTION public.ueberdb_insert_or_update_jsonb(character varying, jsonb) RETURNS void
LANGUAGE plpgsql
AS $_$ BEGIN IF EXISTS( SELECT * FROM store WHERE key = $1 ) THEN UPDATE store SET value = $2 WHERE key = $1; ELSE INSERT INTO store(key,value) VALUES( $1, $2 ); END IF; RETURN; END; $_$;
ALTER FUNCTION public.ueberdb_insert_or_update_jsonb(character varying, jsonb) OWNER TO etherpad;
SET default_tablespace = '';
SET default_with_oids = false;
CREATE TABLE public.store (
key character varying(100) NOT NULL,
value jsonb NOT NULL
);
ALTER TABLE public.store OWNER TO etherpad;
ALTER TABLE ONLY public.store
ADD CONSTRAINT store_pkey PRIMARY KEY (key);
I took the rollback road. I tried to rollback just etherpad to 1.8.4 but it didn't work:
[2020-09-11 20:55:44.494] [ERROR] console - JSON-PROBLEM:[object Object]
[2020-09-11 20:55:44.495] [ERROR] console - (node:1) UnhandledPromiseRejectionWarning: SyntaxError: Unexpected token o in JSON at position 1
at JSON.parse (<anonymous>)
at /opt/etherpad-lite/src/node_modules/ueberdb2/CacheAndBufferLayer.js:169:24
at Query.<anonymous> (/opt/etherpad-lite/src/node_modules/ueberdb2/postgres_common.js:99:5)
at Query.handleReadyForQuery (/opt/etherpad-lite/src/node_modules/pg/lib/query.js:128:10)
at Connection.<anonymous> (/opt/etherpad-lite/src/node_modules/pg/lib/client.js:247:19)
at Connection.emit (events.js:198:13)
at Connection.EventEmitter.emit (domain.js:448:20)
at Socket.<anonymous> (/opt/etherpad-lite/src/node_modules/pg/lib/connection.js:134:12)
at Socket.emit (events.js:198:13)
at Socket.EventEmitter.emit (domain.js:448:20)
[2020-09-11 20:55:44.496] [ERROR] console - (node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
[2020-09-11 20:55:44.496] [ERROR] console - (node:1) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
So I restored a pgsql dump as well and now 1.8.4 works again (which is not really a surprise since it had been working finely).
reason: from 1.8.4 to 1.8.5 the column value is changed from type text to jsonb.
Should be a major release...
Is there a migration script which can be applied?
I'm just gonna revert these commits. This was all poorly handled IMHO.
The change log says "IMPORTANT: This fixes a severe problem with postgresql in 1.8.5" for 1.8.6, but I've upgraded and still get the exceptions mentioned above. I now noticed these exceptions as well, which are output immediately after service restart:
[2020-09-19 17:47:45.482] [ERROR] console - JSON-PROBLEM:[object Object]
[2020-09-19 17:47:45.496] [ERROR] console - (node:30613) UnhandledPromiseRejectionWarning: SyntaxError: Unexpected token o in JSON at position 1
at JSON.parse (<anonymous>)
at /usr/local/var/etherpad/repo/src/node_modules/ueberdb2/lib/CacheAndBufferLayer.js:169:24
at Query.<anonymous> (/usr/local/var/etherpad/repo/src/node_modules/ueberdb2/databases/postgres_common.js:99:5)
at Query.handleReadyForQuery (/usr/local/var/etherpad/repo/src/node_modules/pg/lib/query.js:122:12)
at Client._handleReadyForQuery (/usr/local/var/etherpad/repo/src/node_modules/pg/lib/client.js:281:19)
at Connection.emit (events.js:315:20)
at Connection.EventEmitter.emit (domain.js:483:12)
at /usr/local/var/etherpad/repo/src/node_modules/pg/lib/connection.js:109:12
at Parser.parse (/usr/local/var/etherpad/repo/src/node_modules/pg-protocol/dist/parser.js:40:17)
at Socket.<anonymous> (/usr/local/var/etherpad/repo/src/node_modules/pg-protocol/dist/index.js:8:42)
[2020-09-19 17:47:45.497] [ERROR] console - (node:30613) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
[2020-09-19 17:47:45.497] [ERROR] console - (node:30613) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
@c7hm4r did your database change or something? If you start a new tarble/database are things okay?
@JohnMcLear When restart Etherpad 1.8.6 with an empty database, a new schema is created and it seems to work without exceptions.
When I compare the new schema with the old one, I get following:
diff --git a/etherpad_schema.sql b/etherpad_schema_new.sql
index 9baf5f4..ad3c08b 100644
--- a/etherpad_schema.sql
+++ b/etherpad_schema_new.sql
@@ -41,17 +41,6 @@ CREATE FUNCTION public.ueberdb_insert_or_update(character varying, text) RETURNS
ALTER FUNCTION public.ueberdb_insert_or_update(character varying, text) OWNER TO etherpad;
---
--- Name: ueberdb_insert_or_update_jsonb(character varying, jsonb); Type: FUNCTION; Schema: public; Owner: etherpad
---
-
-CREATE FUNCTION public.ueberdb_insert_or_update_jsonb(character varying, jsonb) RETURNS void
- LANGUAGE plpgsql
- AS $_$ BEGIN IF EXISTS( SELECT * FROM store WHERE key = $1 ) THEN UPDATE store SET value = $2 WHERE key = $1; ELSE INSERT INTO store(key,value) VALUES( $1, $2 ); END IF; RETURN; END; $_$;
-
-
-ALTER FUNCTION public.ueberdb_insert_or_update_jsonb(character varying, jsonb) OWNER TO etherpad;
-
SET default_tablespace = '';
SET default_with_oids = false;
@@ -62,7 +51,7 @@ SET default_with_oids = false;
CREATE TABLE public.store (
key character varying(100) NOT NULL,
- value jsonb NOT NULL
+ value text NOT NULL
);
So the question is: When the type of the column has changed and how to migrate/revert that?
I configured automatic updates for my Etherpad instance from the master branch.
@c7hm4r @qsuscs was the author of the offending commits. I'm hoping he takes some responsibility here and helps out :)
https://github.com/ether/ueberDB/commit/b9e786e2c8753f4cbbe09d3695a92a1a03904ead#diff-f1b8245dd259fa912c33cb2db527f609R31 is the alter statement..
var updateValueColumnType = "ALTER TABLE store " +
"ALTER COLUMN value " +
"TYPE jsonb " +
"USING value::jsonb";
One would assume changing jsonb to text would do it?
Thanks @JohnMcLear , running the following made it work again:
alter table "store" alter column "value" type text;
@c7hm4r no worries, hopefully the handful of people this affects will find this thread and do that command., Thanks for writing and testing! :) I'd be willing to take a PR to fix if you or the original author wanted. I have other tasks at the moment that are taking priority.
Most helpful comment
Thanks @JohnMcLear , running the following made it work again: