Laverna: md batch import with bash script

Created on 20 Jul 2016  ·  14Comments  ·  Source: Laverna/laverna

Hi,
using geeknote I was able to export all my notes in md files (one per note), each notebook has its own folder on my FS, each folder name is the same of corresponding notebook. I thought an abstract on how to implement the import in laverna by a bash shell script and I would like to ask you some questions to understand if the procedure I have in mind could work :

  • is the encrypted part of json note file in md format?
  • which is the encryption method? What about parameters needed to encrypt/decrypt?
  • which are json note mandatory fields?
    The procedure should be something like this :
  • in laverna export data to get a full import/export file
  • unzip the file
  • for each note in md crypt the content
  • create one json file per note with the encrypted version of md file, using for file number a number not already taken by existing notes, and using folder name to determine notebook
  • put the new created json files into notes folder
  • recreate the zip and import back into laverna

Could this work? Any suggestion or advice?

Obviously, if I can get something working I will provide you the full script, hoping it will be useful to other users.

Let me know

Best regards
magowiz

question

Most helpful comment

Ok I did it, I read again the initial indication and I removed all metadata from json files that was not essential.
Now as promised I will publish the scripts (one for getting notes using geeknote, one for building laverna import file) I hope my work will be appreciated .

Warning

This script may destroy your data as well as it may work flawlessly, please, please, please make a backup of all your notes (evernote and laverna ones) , it is better you first try it in another profile or in another dir than usual laverna data one or please backup the whole ~/.config/laverna directory

If you find a bug or you have an idea to improve it, please write me here on github.

Prerequisites :

I used a few packages other than simply bash v4, anyway I will list them to give you the possibility to run the scripts :

  • geeknote : to retrieve all your notes from evernote and save them in md format
  • uuid-runtime package from which I used uuidgen to generate uuid for notes/notebooks
  • jq : I used it to generate json files , get info from them or modify existing ones
  • any package(s) you prefer that ships zip and unzip commands
  • an exported data file from laverna

Usage

  • Do you remember I told you to backup? .... Backup :)
  • you first need to create an empty folder for saving evernote exported files
  • launch evernote-back.sh script with first and only argument the path to your just created folder
  • the script will download every note, it will create one folder for each notebook and put them in the right place
  • due to evernote service limitation, if you have several notes it may happen that the script takes so much time, it is because there is a limit on how many notes you can get per time, after that you have to wait an hour to be able to download again
  • but don't worry, the script can handle this : when evernote tells the limit it's reached it will wait for the necessary time to continue : just be very patient and let it go....
  • once evernote-back has finished you will have all your notes in md format
  • you can then launch lvn-md-import.sh with 2 arguments : first one is the path in wich you have your evernote exported md, second one is files the path to your laverna exported data.
  • once finished you will have a new laverna import/export file (zip) into to-import folder, with all new notes and notebooks from evernote together with laverna already existing data
  • now you can import new zip file into laverna, using import data feature.

UPDATE :

previous version of evernote backup script was bugged about log file location, now the log file is in same folder of exported data, I reattached the zip with evernote-back script updated, please download it again if you downloaded it previously.
lv-md-import.zip

All 14 comments

Hi,

  • Files in exported zip file are not encrypted which means you don't have to encrypt them
  • AES.
  • The only mandatory fields are probably title and ID. So you can provide just ID, title, and content

It should work because there is nothing special about the import process. The structure of the folder should be somethings likes this:

./
└── notes-db
    ├── configs.json
    ├── notebooks.json
    ├── notes
    │   ├── note-id-1.json
    │   ├── note-id-1.md
    │   ├── note-id-2.json
    │   ├── note-id-2.md

As you can see from this tree, only notes have separate files for each item. For instance, all notebooks should be saved in file named notebooks.json.

Hi wwebfor,
first of all thank you for the information provided, I was able today to try to import, manually one md note into laverna without issues, this test was a prerequisite about what I'm going to write in my code.

The whole process is simpler than I thought, I didn't notice in first instance that note content is saved in another file , in md format file, so I thought I had to extract content from json files.

I will obviously provide you the code I will write.

I'm still working on it, in particular to notes json files, I noticed that in each json there is encryptData element that contains an inner object, I think it is the information laverna needs to encrypt back once imported,the content of that element should be identical on each json. Are my thoughts correct? Is it safe to use data from an existing laverna note to create json of the imported note? For what I understood it should be safe to let it empty.
Which criterias, formats or algorithms does laverna use to determine notes filenames or notebook Ids? At some point I will need to give names to md and json files and id to the new notebooks.
Thanks in advance.
magowiz

Ok I wrote the script and I generated the new zip file from an export using the same structure as described on first reply : anyway once I choose to import it does nothing, the same notes as before are there.
I'm pretty sure that every note has an unique filename in uuid format, each note refers to its notebook and every new notebook are inserted into notebook.json file . I don't know what I've done wrong, can you please tell me how to "debug" the importing process or tell me if you want the script I wrote so you can test a generated zip file.
Thanks
magowiz

Ok I did it, I read again the initial indication and I removed all metadata from json files that was not essential.
Now as promised I will publish the scripts (one for getting notes using geeknote, one for building laverna import file) I hope my work will be appreciated .

Warning

This script may destroy your data as well as it may work flawlessly, please, please, please make a backup of all your notes (evernote and laverna ones) , it is better you first try it in another profile or in another dir than usual laverna data one or please backup the whole ~/.config/laverna directory

If you find a bug or you have an idea to improve it, please write me here on github.

Prerequisites :

I used a few packages other than simply bash v4, anyway I will list them to give you the possibility to run the scripts :

  • geeknote : to retrieve all your notes from evernote and save them in md format
  • uuid-runtime package from which I used uuidgen to generate uuid for notes/notebooks
  • jq : I used it to generate json files , get info from them or modify existing ones
  • any package(s) you prefer that ships zip and unzip commands
  • an exported data file from laverna

Usage

  • Do you remember I told you to backup? .... Backup :)
  • you first need to create an empty folder for saving evernote exported files
  • launch evernote-back.sh script with first and only argument the path to your just created folder
  • the script will download every note, it will create one folder for each notebook and put them in the right place
  • due to evernote service limitation, if you have several notes it may happen that the script takes so much time, it is because there is a limit on how many notes you can get per time, after that you have to wait an hour to be able to download again
  • but don't worry, the script can handle this : when evernote tells the limit it's reached it will wait for the necessary time to continue : just be very patient and let it go....
  • once evernote-back has finished you will have all your notes in md format
  • you can then launch lvn-md-import.sh with 2 arguments : first one is the path in wich you have your evernote exported md, second one is files the path to your laverna exported data.
  • once finished you will have a new laverna import/export file (zip) into to-import folder, with all new notes and notebooks from evernote together with laverna already existing data
  • now you can import new zip file into laverna, using import data feature.

UPDATE :

previous version of evernote backup script was bugged about log file location, now the log file is in same folder of exported data, I reattached the zip with evernote-back script updated, please download it again if you downloaded it previously.
lv-md-import.zip

Sounds nice. Although I am not an Evernote user :)

the second script isn't so related to evernote, but to every note program from wich you can export to md format. ;)

Ah ok, did not read carefully.

No problem at all, for me all it started with the need to migrate from evernote :-) Then wwebfor gave me some precious information about laverna file format and I tried to get something working.

I read the instruction I gave, I choose to isolate what's needed if you don't need to migrate from evernote:

  • you don't need to install geeknote : nothing from evernote to retrieve
  • at some point you have to save your exported notes somewhere , remember that from original path every folder will be translated to notebook with same name which contains the notes in it , for example if you tell to script lvn-md-import a directory named /home/user/notes , and if you have in notes directory a folder named test and some md file notes in it, all that notes will be assigned to test notebook
  • every md file needs to have note title as filename
  • you need to export data from laverna
  • start the original usage instructions from launching lvn-md-import.sh

That's it, the only thing is that you have to find yourself a method to get notes in md files from program you need to migrate from..... .
I hope I clarified better the whole process.

This... is awesome. I've dreamt for so long to have something similar to escape away from Evernote (with my content).

Except i have several questions for you before i dig into it : I've been a long time evernote user and to this day have ~1700 notes. So here are my concerns :

  • how about evernote tags ? They are the main way to organise notes in a good searchable way along with a solid thesaurus. Do your script adds those tags ?
  • you wrote about a cooldown between queries from evernote. What amount of time would be necessary to import all the 1700 notes ?

Anyway, thanks for your job on this, can't wait to fully migrate :)

Hi Julianoe I appreciate your enthusiasm about what I've done, that is, keep in mind, nothing more than putting together the information wwebfor gave to me, the geeknote program , and some tries to get something working. Nothing more than that
Now I answer to your questions :

  • Mine script, currently, doesn't support evernote tags, I have to say that I don't know if geeknote can handle them, since the output it gives is pure md files. I have to investigate about geeknote usage and check if there is a way to retrieve note tags. If this is possible, I will add this information on laverna json files .
  • About rate limit, I did not find a specification about how much you have to wait, I tried a new export and I noticed that after 171 notes I hit the limit and I have to wait 1941 seconds (about 30 minutes) , so you can simply, having this data, estimate how much time you need for 1700 notes .

I hope my answer satisfies you.
I noticed some issues on log file of evernote script, I fixed it and I will update the one I attached here.

A small update on tags : I read geeknote documentation and for what I understood there is a way to list all tags, using command : geeknote tag-list , anyway I didn't find a way using gnsync or other geeknote commands to get a list of notes from a given tag or to get a list of tags from a given note.
This makes almost useless the tag synchronization, since you will get all tags you had in evernote but you lost the association between them and notes. I think if it is so it wouldn't be worth to implement such feature in the script.
I hope I am wrong, anyway you can read geeknote documentation and tell me if there is a way to do it that I didn't find.

UPDATE : Nevermind I didn't search in right places, geeknote find command it's the one we can use, in particular with geeknote find --tags, I will do some tries and let you know.... Now tags migration makes sense.

@magowiz thank you for scripts, since geeknote not always relevant due to Evernote API limitations, here's some mix of scripts that worked https://github.com/m00rt1/enex2laverna

Was this page helpful?
0 / 5 - 0 ratings

Related issues

Issam2204 picture Issam2204  ·  8Comments

inukaze picture inukaze  ·  9Comments

JerJohn15 picture JerJohn15  ·  4Comments

valvin1 picture valvin1  ·  3Comments

hgaronfolo picture hgaronfolo  ·  5Comments