Etherpad-lite: Uncaught Error: Failed assertion: Invalid changeset (checkRep failed)

Try the latest Develop branch and let us know if it still occurs

This is not trivial as it occurs by chance on the prod server with many users and I cannot reproduce it.
I can only test if broken pads stay broken on develop, if that helps.

yes please

I will get the db copied to dev next week. Will report as soon as possible. Thank you for the fast response.

Unfortunately moving to develop has not repaired the damaged pads. Interestingly, moving servers (simple sql export/import) has "repaired" one of the pads. It works on the new server (even on 1.3.0) but other damages pads don't. Still the same error.

This is a really strange bug and pads seem to sometimes just "self-repair" even on PROD and with no change whatsoever from us.

In theory this problem shouldn't occur at all as the bad data should never find it's way in now..

I can leave this open and if it occurs on fresh data we can try to resolve it..

What do you mean with "fresh data" ? Do I need to put PROD on develop and try to get new broken pads?

Hrm, that'd be painful potentially.. Maybe wait for 1.4 which should be out in a few weeks max.

We might do that. Thanks a lot.

I also have that problem, with the same strange randomness. I updated to etherpad-lite 1.4 and it is still there. URL for one of the pads http://etherpad.usabilidoido.com.br:8080/p/07318a9b2b5666637d870fc50656323620af4df4

I am right now in the process of upgrading to 1.4 and hopefully going live with the new version, so I can check if new defects come up in fresh pads after running on the new version.

@usabilidoido you can tell your users to export-import the pad. You can access controls by adding /timeslider to the url like this: http://etherpad.usabilidoido.com.br:8080/p/07318a9b2b5666637d870fc50656323620af4df4/timeslider

Export as html and then import into a new pad.

I am experiencing this issue in 1.4. On broken pads, command line shows [WARN] client - Uncaught TypeError: Cannot read property 'length' of undefined when browser shows the Failed assertion error.

Hat tip to @Ra1d3n for the /timeslider workaround. Glad to see pad content is still accessible there.

This is just a hunch, but if you like, try with the experimental try/client-init-remove-checkRep branch I just created. (This is generally a dangerous thing to do, but it's worth a try, I think.)

I have upgraded everything to 1.4. and yesterday we had a broken pad again. Might still be one that broke before update, but I am not sure. I will keep on looking.

@marcelklehr Unfortunately, I can't move production server to a _dangerous_ version. And I can't mirror requests to a secondary server because I don't have the resources. :-/

Sorry, I wasn't clear: Don't run try/client-init-remove-checkRep in production, but try to access the broken pads with etherpad running on that branch.

I removed checkRep in that branch, because I suspect that normalization may not work correctly in some cases. So, when a broken pad works on that branch without any problems at all, we've got to revisit this method.

@marcelklehr I just did, and unfortunately it did not help.

Process:

git fetch
git checkout try/client-init-remove-checkRep
git status
On branch try/client-init-remove-checkRep
Your branch is up-to-date with 'origin/try/client-init-remove-checkRep'.

I confirmed the change is actually in the file system. (comment and new line are there) Error is still the same.

I did get the error that @ericpedia mentioned, and it did not occur on pads other than the corrupted one.

console on server:

[2014-05-09 16:55:39.152] [WARN] client - Uncaught TypeError: Cannot read property 'length' of undefined -- { errorId: 'dTtndCRA5gonLZyvMlqw',
  msg: 'Uncaught TypeError: Cannot read property \'length\' of undefined',
  url: 'http://localhost:9001/p/OkTJWMYVNs',
  linenumber: 15,
  userAgent: 'Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.131 Safari/537.36' }

When I kill the server the message is displayed in the client, also:

I might be able to give you an SQL import of the broken pad, but you would need to tell me what exactly you need extracted from the database first. :-)

The actual pad content would be very helpful, so that'd be db key pad:<PADID> (http://etherpad.org/doc/v1.4.0/#index_pad_padid) and perhaps a few of the last revisions.

I think I'm getting closer:

checkPad says:

$ bin/checkPad <padID>
[WARN] console - DirtyDB is used. This is fine for testing but not recommended for production.
[ERROR] console - Bad changeset at revision 4901 - Failed assertion: mismatched apply: 11636 / 11635
[ERROR] console - Bad changeset at revision 11401 - Failed assertion: mismatched apply: 42094 / 42093
[ERROR] console - Bad changeset at revision 12301 - Failed assertion: mismatched apply: 48875 / 48874
[ERROR] console - Bad changeset at revision 13601 - Failed assertion: mismatched apply: 60227 / 60226
[INFO] console - finished

Mismatched apply in all cases means that the changeset in question expected one character less than actually present (so, one unexpected character). After examining the db dump, I found out that all of these changesets follow a revision with an atext attached to it (not all revisions include the full pad contents, but instead just the delta, however, every so often for performance reasons the full contents are attached).

Presumably, something went awry with this meta property either upon storing the revision or upon creating the current atext for a new pad client.

I just rewrote checkPad to use the computed atext instead of the cached one and the pad passes. Even the cached atexts are correct! This makes me think there's a bug in some algorithm that computes the full atext for the client_vars!

Nice work so far Marcel. Sounds like you are close to solving this.

Ah, it's not the client_vars generator. The algorithm responsible for creating the cached atext is broken.

@Ra1d3n As a quickfix, you can revive your broken pads by deleting the atext meta property from all key revisions (where revNo % 100 == 0). "Re-inserting" all records related to a broken pad was reported as a fix for similar problems a long time ago -- now we know why it works :)

@marcelklehr Will that make EP rebuild the pad from revision 0 on load? I guess I should not go around deleting all atext properties from all revisions then... :-) Any hope for a "regenerate Pad" script that rebuilds key revisions with a correct algorithm?

I've made some changes to the checkPad-script. See here: #1653
But this is more a dirty hack. If my script will resolve this problem I will write a script to fix pads

Cool, I look forward to it. Right now we get only about one broken pad a week, so I am inclined to wait for a proper fix. :-) Thanks.

Yea, I was thinking of a script.

So, the diff between the computed atext and the cached one shows that: "концертов" somehow got turned into "ко��цертов". In another revision, "з" was turned into "��" as well...

I'm not sure what this is about. These chars are in the Unicode BMP, afaik, so I don't know what happened to them.

The mutations in @Ra1d3n's pad occur in key revisions 4900, 11400, 12300 and 13600 (every 100th rev is a key rev, meaning it'll cache the full pad contents). Also, the AttributedText stored in the pad record is corrupt, too. All other key revisions are fine. (analyzed with this script)

The chars seem offly random. Nothing sticks out, really. I don't see a pattern.

I suspect there's something going wrong when storing the AttributedText in the database. Since sometimes the pad recovers in between broken key revisions, I'm guessing that the atext stored in memory is good. Sometimes, when it is stored in the db it somehow gets corrupted, though. If authors continue to edit the document until the next key revision is created and that revision doesn't end up getting corrupted, then nobody will notice anything.
However, if the server shuts down _before_ managing to successfully save the valid AText held in memory to the database, the pad will be broken on the next server start.

@marcelklehr We had etherpad crash and recover a few times because of a plugin that was not well coded, could this be the cause? Strange that it's just one letter that got corrupted every time.

I created a script to reinsert all database value of a pad. On my broken pads the script hasn't worked.
@Ra1d3n You can download the script here: https://github.com/Gared/etherpad-lite/blob/pad-repair-script/bin/repairPad.js
Make sure your etherpad instance isn't running while you're executing this script.

@Gared Where do you get the values from in your script? I recommend working off my checkPad fork, tweaking it to insert the re-computed AttributedText values into the database.

@Ra1d3n Etherpad crashes are unlikely to cause data corruption, I'd say.

@marcelklehr It probably just made the problem more visible, because the correct memory value was lost for us in those cases.

@marcelklehr This script is a fork from the extractPadData script. The values and keys are loaded in the function above. These are all keys of a pad.

The repairPad.js script from @Gared is fixing my broken pads. Any chance of this fix being incorporated in the next etherpad-lite version?

Sure just send a pull request with it in or ask @Gared to

Opened pull request #2210

My etherpad is running on 1.4.1 and I have got 3 times the same problem described above : the pad is not able to load but /timeslider works well.
2 of them are now rerunning without doing nothing.
On the third pad, I tried unsuccessfully the repairPad.js. Its url is :+1: http://portail.univ-lille1.fr/etherpad/link.jsp?groupID=g.jfobkKVrkydeTwLY&padID=SES_Grp8_Macroeconomie (you ave to click on "utilisez le pad avec l'invitexy" to access to the paditself.

Maybe there is a changeset with an anormal value which is not taken into account by repairPad or is there any new release of reparPad.js I have not seen on git ?

We think we have a fix for this now. Please pull develop and test :) Thanks!

Hello, we 've just had this occur. We run 1.5.7, that is the last released version. Backend is a MySQL database. I have no indications as to user actions that might have caused this.

Pad in question: https://etherpad.wikimedia.org/p/iOS-iteration-planning

Getting the pad data out via the timeslider trick works fine. The pad however will never load.

Anything in the database can be dumped and provided for debugging if it helps.

hi, I had this discussion in the morning.
08:47 < webzwo0i> mutante: i debugged the pad. you normally should not do this, but if you have a database backup (and after making export/etherpad to have a backup of
the pad) you can change three database entries and the pad should be working fine again. the three mysql commands are

mysql> update `store` set `value` = replace(`value`,'????','??') where `key` like "pad:iOS-iteration-planning";
mysql> update `store` set `value` = replace(`value`,'????','??') where `key` like "pad:iOS-iteration-planning:revs:7200";
mysql> update `store` set `value` = replace(`value`,'????','??') where `key` like "pad:iOS-iteration-planning:revs:7300";

08:49 < webzwo0i> can you check if your database is running utf8mb4 charset or utf8?
08:51 < webzwo0i> oha ups please dont apply the mysql-commands. maybe i was a little bit to fast :-) need to check something first
09:08 < webzwo0i> mh nope should be fine, please test... it would be good to know if you ran latest release or something else and which plugins you have enabled
09:47 < webzwo0i> I don't know if you ppl at wikimedia know each other but if you can find out who the user "Brian" is could you ask him what browser he is using? the
reason is I can see what the bug is, but I cannot trigger it in my browser (only manually, but because you ppl are not hostile it was probably not on
purpose)
09:49 < webzwo0i> (so we probably have two bugs, one server-side and one client-side)

the info I need is if your database is utf8 or utf8mb4
I have extracted the offending changeset, the timeslider will not work around these revisions if you don't also apply

mysql> update `store` set `value` = replace(`value`,'????','??') where `key` like "pad:iOS-iteration-planning:revs:7105";

together with the updates from above this should make your pad reusable again

@webzwo0i

Yeah, I noticed that mutante talked to you. I just decided to follow up on the github issue as well. I 'll track down who "Brian" is and what browser they are using for you.

So, the store table is utf8mb4 for quite some time now. It was utf8 but we converted to utf8mb4 due to a different set of problems back in Jun. Specifically on Jun 23, 2015

https://github.com/ether/etherpad-lite/issues/2522#issuecomment-114441189 ;-)

no need to find out the user/browser, I can reproduce the bug now. thank you!

Because you are on latest release you need to insert "charset":"utf8mb4" into settings.json inside dbSettings. This is now in settings.json.template. You can check if this is necessary with

SHOW VARIABLES LIKE 'character_set%';
SHOW VARIABLES LIKE 'collation%';

client (and maybe connection?) should indicate utf8mb4, if not your database-tables itself are capable of storing 4 byte utf8 but the server does not expect 4byte utf8 an the client-connection.
This does not repair your old pads. You can however iterate over all your pads and use bin/checkPad.js to get an idea how many and which pads might have similar problems. Depending on the circumstances it can be very easy to repair (although some characters will be broken) like in the above case. If there are lots of broken pads it might make sense to automate this.

The reason these problems are not seen when people actually are writing is that most sites use ueberDB's internal cache for performance. This pure-javascript cache is totally aware of Unicode. As soon as the cache is flushed or etherpad is restarted it needs to get the entries from the database.

I 've repaired the pad as you 've instructed. Interesting to find out that 4 question marks in a row would corrupt a PAD in such a way. And that the corrupting changeset would be so old compared to the tip of the pad. But your explanation makes sense, thanks for that.

I 've updated the configuration with "charset": "utf8mb4" as well.

I am following the phabricator ticket at wikimedia but don't have an account there so I post it here.

Your second broken pad can also be repaired using:

update `store` set `value` = replace(`value`,'????','??') where `key` like "pad:iOS_Retrospectives:revs:1120";
update `store` set `value` = replace(`value`,'????','??') where `key` like "pad:iOS_Retrospectives:revs:1254";
update `store` set `value` = replace(`value`,'????','??') where `key` like "pad:iOS_Retrospectives:revs:1216";
update `store` set `value` = replace(`value`,'????','??') where `key` like "pad:iOS_Retrospectives:revs:1108";
update `store` set `value` = replace(`value`,'????','??') where `key` like "pad:iOS_Retrospectives:revs:1106";
update `store` set `value` = replace(`value`,'????','??') where `key` like "pad:iOS_Retrospectives:revs:1200";
update `store` set `value` = replace(`value`,'????','??') where `key` like "pad:iOS_Retrospectives:revs:1300";
update `store` set `value` = replace(`value`,'????','??') where `key` like "pad:iOS_Retrospectives:revs:1400";
update `store` set `value` = replace(`value`,'????','??') where `key` like "pad:iOS_Retrospectives:revs:1500";
update `store` set `value` = replace(`value`,'????','??') where `key` like "pad:iOS_Retrospectives:revs:1600";
update `store` set `value` = replace(`value`,'????','??') where `key` like "pad:iOS_Retrospectives:revs:1700";
update `store` set `value` = replace(`value`,'????','??') where `key` like "pad:iOS_Retrospectives:revs:1800";
update `store` set `value` = replace(`value`,'????','??') where `key` like "pad:iOS_Retrospectives:revs:1900";
update `store` set `value` = replace(`value`,'????','??') where `key` like "pad:iOS_Retrospectives:revs:2000";
update `store` set `value` = replace(`value`,'????','??') where `key` like "pad:iOS_Retrospectives:revs:2100";
update `store` set `value` = replace(`value`,'????','??') where `key` like "pad:iOS_Retrospectives:revs:2200";
update `store` set `value` = replace(`value`,'????','??') where `key` like "pad:iOS_Retrospectives:revs:2300";
update `store` set `value` = replace(`value`,'????','??') where `key` like "pad:iOS_Retrospectives:revs:2400";
update `store` set `value` = replace(`value`,'????','??') where `key` like "pad:iOS_Retrospectives";

The 4 question marks are the symptom because, iirc, the single bytes in four-byte UTF8 are not valid UTF8. (In UTF8 only the first 127 chars are represented as single bytes, multibyte UTF8 probably uses bytes above 0x7f). So 4 question marks actually represent one 4byte encoded UTF8 string, which represents an code point outside the Basic Multilingual Plane (most probably an emoji :-D). In Javascript those code points would be encoded using UTF16's surrogate pairs, which are 2 16bit-values.

The checkRep-problem is that in changesets we not only store the characters but also the length of the change. Javascript's length() function, however, counts surrogate pairs as 2, so e.g. an emoji has length 2. When mysql decodes the string of a changeset to question marks than our representation of a changeset is not valid anymore.

Replacing it with two question marks is a hack not a real solution because we have no idea which code point the user entered in the first place (but as long as the value is in ueberDB's cache we could get it out from there).

it would produce wrong results if someone actually uses four question marks (our length value would indicate 4, if we replace it with 2 question marks we would get a checkRep error in return) so if we would automate a repair script we would need to check if the string length after applying our change is conforming to the "how many chars are added"-value of the changeset.

to circument the problem if someone actually used four question marks and additionally code points like emoji we also need to track the position inside the document where we replace the question marks.

also note that not every checkRep-error is caused by broken encoding

And of course the above worked. AWESOME! I can not thank you enough. I am hoping that with the configuration fix in place this wont happen again.

I was wondering if the debugging you are doing is manually btw. Getting and comparing the changesets and their lengths one by one manually or if you have some more automated way of doing that. I suppose I can create one of my own anyway, just being curious

<3 @webzwo0i Thanks man you are awesome

@webzwo0i I have been doing a fair amount w/ getting the char rep from a X co-ord on an element lately (for drag and drop) and I have a feeling I'm going to hit this problem where certain characters are improperly wrapped. It will be interesting to test that at some point

The error can easily be reproduced by creating a new pad with a single emoji (e.g. 🐼) and restarting etherpad, see also #3340.

Update: As of April 2019, this single emoji itself doesn't break a pad, even after restarting.

I wanted to check all pads, so I added a checkAllPads tool (see PR #3342).

The error can easily be reproduced by creating a new pad with a single emoji (e.g. panda_face) and restarting etherpad, see also #3340.

I can not reproduce this, doing exactly what described above. See https://etherpad.wikimedia.org/p/ohmy for an example (yes I 've restarted etherpad multiple times already)

We just had a pad break with this error as well. Curiously, checkPad,js does not find any problem, and repairPad.js runs to completion without fixing it. Is there any way to determine which revision is at fault?

EDIT: Ah, I found https://gist.github.com/marcelklehr/a78d293571e7f06e3cf9 which pointed me the right way. Any chance this could be included in etherpad itself? It has been infinitely helpful right now, thanks a ton! (However, I had to replace console.log by console.error to even see any revision numbers. I have no nodeJS experience whatsoever, but I couldn't figure out another way to actually see all the logging.)

Indeed doing the "replace ???? by ??" helped here as well. :) Seems like the last changeset was someone inserting an emoji (it ended in $????).

However, I do not understand why this is classified as a "minor bug". This bug leads to total loss of a pad (until someone notices the /timeslider thing, which took a week in our case, and even then history is lost).

Unassigned myself, as it's unlikely I'll get to fixing this. FWIW, this bug appears to be due a limitation of the easysync library, which I'm speculating does not to support all of utf-8. (UTF-8 may encode one character as multiple bytes, which each add to the length of a string in javascript, even though it's just one character.)

-- nevermind -- :D

FWIW, this bug appears to be due a limitation of the easysync library, which I'm speculating does not to support all of utf-8. (UTF-8 may encode one character as multiple bytes, which each add to the length of a string in javascript, even though it's just one character.)

Actually we have umlauts (äöü) in our pads all the time, which are also multi-byte in UTF-8. Based on what has been said above, I think the issue is actually about UTF-16 -- which, when originally designed, was intended to have exactly 2 bytes per character (codepoint, really), but now that we have more than 2^16 codepoints there are some that need 4 bytes, like emojis. And now length() no longer matches the number of codepoints, and everything gets confused.

So maybe a better fix is to outright reject any surrogate pairs (4-byte codepoints)? That would make it impossible to use etherpad with characters from the supplementary plane, but that's likely broken anyways it seems? And it should protect the DB. There seem to be ways to test for surrogate pairs in JS (but I have zero experience in modern JavaScript).

Why did this get closed? To my knowledge, Etherpad still chokes on characters outside the BMP. I recently again had to manually repair a pad that got broken this way.

I closed it because I opened the Issue 2014 and was not interested in it anymore.

Well, it is still an open problem for others, so I'd appreciate if you could reopen.

Thanks! :)

Does anybody have any example for a character (sequence) that breaks a pad reliably? This would facilitate debugging I guess.

The Easysync library describes text (and its legth) in terms of "characters", but it was a minimum viable product from 10 years ago. Nowadays we should probably think in terms of NFC-normalized UTF-8 code points.

Just wondering, might we be able to solve the problem by storing the ueberdb values as binary blobs rather than in a collated text column?

Currently, if we try to put a byte sequence that is not valid utf8mb4 (think: a changeset that contains part of a multibyte character) into a utf8mb4 column, there are only two possible outcomes: either the database refuses the input, or client (or server) need to remove (think: replace with "?") the invalid "characters" or bytes before.

By using a binary blob column, the database would no longer care about the byte sequence being invalid utf8mb4, so we might avoid the character replacement. If easysync is as encoding agnostic as I understand, this could work (as long as two users don't insert multibyte characters AB and CD at the same position concurrently and these end up as individual changeset A, C, B, D - in this order -, rendering the merged result invalid utf8mb4).

PS: I just tested that inserting a 4-byte UTF8 character like 🍰 is not a problem itself (although: I didn't restart yet, which may be explanation), so I assume the bug either requires concurrency (leading to the character being split up in two or more changesets that are invalid on their own) or it requires a client emitting a changeset that removes part of such a character.

Hi, we are also experiencing this problem on a lot of pads.

I'm trying everything and just can't replace this with 🍰, I tried restarts, different database backends (that are properly configured)..

Can anyone provide steps to replicate with our more modern code base?

Hitting backspace on 🍰 does replace the item with � which is obviously sucky.

For me, replace(value,'????','??') has always worked so far. Hasn't happened for a few months though.

I included an updated version of Check Pad Deltas that works, if people can give that a try to see if it helps when experiencing this problem I'd appreciate it.

I still think the basic problem is that Etherpad data model thinks in terms of "characters" and not normalized UTF-8 code points.

Unless we rework the core library this will never be really solved. Obviously, any mitigation is useful. Just saying that there are no _easy_ solutions that are guaranteed to be 100% correct in my opinion.

You'd be surprised just how many editors (and very popular ones with developers) have a similar experience to Etherpad tho. Playing around today I had some crazy experiences.

I included an updated version of Check Pad Deltas that works, if people can give that a try to see if it helps when experiencing this problem I'd appreciate it.

Pulled in in the master branch with #3717 (14ae2ee95094).

Hi, we are having a similar issue with one of our Pads.
@JohnMcLear unfortunately the latest version of checkPadDeltas did not help :/

@gnd do you have a public instance?

Can you hit the padId/export/etherpad url and get the .etherpad file?

Are you running latest develop?

What's your database backend?

So many questions, please provide as much details as possible

@JohnMcLear
Yes, its a public instance: https://pad.xpub.nl/p/CareCircle
Unfortunately i get a 502 Bad Gateway error trying to get the .etherpad file
We are running latest develop (git pull origin) on nodejs 12.16.3-1nodesource1, with the db backend being 10.3.22-MariaDB-0+deb10u1.

Im available today to help you with any sort of debugging you might want to do. I have already tried the last version of checkPadDeltas, however it just hangs for hours after start. This is the only output it gives:

All relative paths will be interpreted relative to the identified Etherpad base dir: /opt/etherpad
[2020-05-05 00:04:12.330] [DEBUG] AbsolutePaths - Relative path "settings.json" can be rewritten to "/opt/etherpad/settings.json"
[2020-05-05 00:04:12.346] [DEBUG] AbsolutePaths - Relative path "credentials.json" can be rewritten to "/opt/etherpad/credentials.json"
settings loaded from: /opt/etherpad/settings.json
No credentials file found in /opt/etherpad/credentials.json. Ignoring.
[2020-05-05 00:04:12.369] [INFO] console - Using skin "no-skin" in dir: /opt/etherpad/src/static/skins/no-skin
[2020-05-05 00:04:12.371] [INFO] console - Session key loaded from: /opt/etherpad/SESSIONKEY.txt
[2020-05-05 00:04:12.541] [ERROR] console - table is not configured with charset utf8 -- This may lead to crashes when certain characters are pasted in pads
[2020-05-05 00:04:12.543] [INFO] console - RowDataPacket { character_set_name: 'utf8mb4' } utf8

Dude, the error is in your log!

[2020-05-05 00:04:12.541] [ERROR] console - table is not configured with charset utf8 -- This may lead to crashes when certain characters are pasted in pads
[2020-05-05 00:04:12.543] [INFO] console - RowDataPacket { character_set_name: 'utf8mb4' } utf8

See: https://github.com/ether/etherpad-lite/issues/3959

@JohnMcLear
our db has

+----------------------------+------------------------+
| DEFAULT_CHARACTER_SET_NAME | DEFAULT_COLLATION_NAME |
+----------------------------+------------------------+
| utf8 | utf8_general_ci |
+----------------------------+------------------------+

While the store table has

+--------------------+
| character_set_name |
+--------------------+
| utf8mb4 |
+--------------------+

So should i convert using
ALTER DATABASEetherpad_lite_dbCHARACTER SET utf8mb4 COLLATE utf8mb4_bin;

?

@JohnMcLear

The misconfiguration was twofold, the database was using utf8 and utf8_general_ci, but also in the settings.json the charset for the database was set as "utf8". Having fixed that all to utf8mb4 still didnt help, and the pad in question doesnt load, and the checkPadDeltas still hangs:

All relative paths will be interpreted relative to the identified Etherpad base dir: /opt/etherpad
[2020-05-05 13:17:43.443] [DEBUG] AbsolutePaths - Relative path "settings.json" can be rewritten to "/opt/etherpad/settings.json"
[2020-05-05 13:17:43.444] [DEBUG] AbsolutePaths - Relative path "credentials.json" can be rewritten to "/opt/etherpad/credentials.json"
settings loaded from: /opt/etherpad/settings.json
No credentials file found in /opt/etherpad/credentials.json. Ignoring.
[2020-05-05 13:17:43.463] [INFO] console - Using skin "no-skin" in dir: /opt/etherpad/src/static/skins/no-skin
[2020-05-05 13:17:43.464] [INFO] console - Session key loaded from: /opt/etherpad/SESSIONKEY.txt

@gnd It's a GiGo problem. Once you have garbage in, it can't be changed. Now all you know is the problem wont appear in the future!

@gnd It's a GiGo problem. Once you have garbage in, it can't be changed. Now all you know is the problem wont appear in the future!

Wouldn't repairPad.js be able fix these broken pads?

Oh hi @caugner - sadly no, repairPad.js generally sucks and doesn't really work. https://github.com/ether/etherpad-lite/blob/develop/bin/repairPad.js#L48

The best thing I can suggest is to pull the atext/text out of the pad and bring it into a new pad.

@gnd I can write you a script to test to try and get the text if you want?

bin/extractPadData.js with a change to output to stdout might be sufficient here.. 2mins I will create an extractPadText.js

@JohnMcLear that would be quite helpful indeed )

Extracting

Use node bin/extractPadData.js $padid
Then cat $padid.db | grep \"text\" | grep revNum | tail -1

The text is the val.atext.text item, you could json parse this at cli.. I will do that next if you need it.. For now do these commands making sure you replace $padid with your PadID

Parsing

sudo apt-get install jq to install jq then cat $padid.db | grep \"text\" | grep revNum | tail -1 | jq .val.atext.text to see just the text.

To write the Pad text to a text file cat $padid.db | grep \"text\" | grep revNum | tail -1 | jq .val.atext.text > $padid.txt

Now you have the pad text you can just put that in a text file and import or or you setText API or whatever...

Lemme know if extraction fails and I will consider another approach.

The extraction is running, however it is quite slow. In the file CareCicle.db I see the latest line at revs:80, while the script already runs for 20m. The pad in questions has over 12k revisions..

Oh man, that sucks.. I guess it can't build the pad object after 80 revisions.. It should only take 30 seconds or so for the script to run.

the last suggestion would be a big one, to dump the entire db and send it to me and then I can write a script to parse out what you need. Alternatively I can try to write a script here but there might be some back & forth to get it working that way.

Hi @JohnMcLear, the script has finally finished. I have no idea why it took so long (almost 40 hrs). Anyway, when looking into it, it seems to me, the whole exercise can be done by selecting the highest revision which is divisible by 100 from the store table and extracting the text from it ? In the future ill do this by hand :) Thanks a lot for your help

Exactly this, but I often get told off by our users when I make the assumption they can perform database queries so I try to avoid it. I think I know why it took so long btw, are you using MySQL @ Etherpad 1.8.3 ?

I'm using the latest master from git (not sure which version that is)

Assuming MySQL it's a known bug that we're due to have the patch land today.

yes sorry, its latest MariaDB - 10.3.22-MariaDB

@JohnMcLear im sorry to spam this ticket, but do you have an issue open for the MySQL patch you mentioned ? I want to see if our performance troubles with etherpad might be resolved by it.. thanks

No but just do npm install [email protected] to fix

Btw the new logic for storing additional atext is in so this should be closed but if people experience an issue please do create a new issue and refer to this one. I want to deal with each individual cause of problem case-by-case with the main goal to create automated logic to restore a pad upon detected corruption in real time. That's the dream as corruption is inevitable.

This is a message for people getting to this recently (when upgrading from older versions of etherpad).

Today I upgraded an etherpad service from 1.6.3 to 1.8.6 (what a change!!!!! congratulations to all developers)

I had problems with one pad, the checkers (checkPad, checkAllPads, etc.) failed to detect it (or I don't know how to run node fine, anyway).

I verified the charset is utf8mb4 in my settings.json (saw last version in settings.json.template).

  "dbType" : "mysql",
  "dbSettings" : {
    "user":     "etherpaduser",
    "host":     "localhost",
    "port":     3306,
    "password": "PASSWORD",
    "database": "etherpad_lite_db",
    "charset":  "utf8mb4"
  },

for case https://pad.example.com/p/my-broken-pad I did:

mysql
update `store` set `value` = replace(`value`,'????','??') where `key` like "pad:my-broken-pad"

and it worked again :tada: :unicorn: :sparkles:

this solution was above (I put a +1 on previous messages with the solution to help find it), but I wanted to have it more clear

I guess one thing we could do here is check for ???? in pad contents and provide a warning that includes a suggested solution. @pedro-nonfree please could you submit a patch to checkPad.js or something then I'd happily merge that :)