Mimic-code: how to run .sh file in concepts folder to build the relevant table?

What operating system are you running? The file is a shell script, and so will only work on Mac OS X or Linux. If you are on Windows, then I'm not sure of the best approach. The SQL files can be run using the psql (shell) program that came with your PostgreSQL install, but there is a regular expression applied to some files to replace BigQuery syntax with PostgreSQL syntax. Probably the simplest way to emulate that on Windows is to just do it manually.

What operating system are you running? The file is a shell script, and so will only work on Mac OS X or Linux. If you are on Windows, then I'm not sure of the best approach. The SQL files can be run using the psql (shell) program that came with your PostgreSQL install, but there is a regular expression applied to some files to replace BigQuery syntax with PostgreSQL syntax. Probably the simplest way to emulate that on Windows is to just do it manually.

Thank you so much! I am using Win7. Yes, I can run the individual .sql file with PostgreSQL. I am wondering whether there is a way to run postgres_make_concepts.sh in cmd or psql to set up all the tables.
By the way, would you like to provide the make_concepts file in .sql format, just like the file in the sepsis-3-mimic code?
OR I wish someone could provide the algorithm for me to run .sh in Windows. It took me a lot of time to google but got nothing.
Thank you!

OK, it's fairly challenging because I use Unix utilities to alter the scripts before passing them to the psql client. The .sh script (1) modifies the SQL files using a regular expression to make them postgres compatible, (2) creates a table with the same name as the file using the script's content, and (3) runs the SQL scripts in the correct order. So you'd have reproduce the steps manually.

Unfortunately because I use the sed utility, there's no way to make it an SQL file as before. You can install sed on Windows using GNU for Windows and use that program to replace the text. It might be possible to get it to run in command prompt if you also install the GNU core utils for Windows.

Alternatively you could try installing postgres in the Windows Subsystem for Linux (WSL) - tutorial here. If you did that then the .sh script would run in the Linux environment.

dear alistair and smartniko
I met with a similar issue.
I have followed the instructions and run the postgres-funtions.sql, yet now i could not run each .sql manually.... Did I do the first step wrong? How could I undo the first step...

Thanks

No the first step is absolutely required, and creates functions on postgresql you'll need. You can test if it worked with:

SELECT REGEXP_EXTRACT('test it grabs the number 123', '[0-9]+');

should return 123. If you get a message No function matches the given name and argument types., then you haven't created the functions on the database.

Once that's done, you need to do the following text replacements in all the SQL files:

1. Replace physionet-data.mimiciii_clinical.<table_name>, physionet-data.mimiciii_derived.<table_name> , and physionet-data.mimiciii_notes.<table_name> with just <table_name>. This is done by the REGEX_SCHEMA call in the shell script.
2. Replace DATETIME_DIFF(date1, date2, DATE_PART) with DATETIME_DIFF(date1, date2, 'DATE_PART'), that is the DATE_PART needs to be quoted with single quotes. This is done by the REGEX_DATETIME_DIFF call in the shell script.
3. Add a create table statement at the top of the file, e.g. if the file is named echo_data.sql, add CREATE TABLE echo_data AS at the top of the file. This is done by the echo calls in the shell script.

The above three steps replicate what is done in the shell script (postgres_make_concepts.sh). Of course automating them is best, so if you know how to do that in Windows, go for it (and post your solution here!).

dear alistair and smartniko
I am a medical student and not very good at SQL/psql. I tried hard to run make_concepts.sh manually but failed. And my system is windows too.
I have created functions on postgresql and they have appeared but couldn't be uesd. (I got 'No function matches the given name and argument types' after SELECT REGEXP_EXTRACT('test it grabs the number 123', '[0-9]+');)
When I copied the sql and tried to create ventilation view it said 'No function matches the given name and argument types', but it was successfully created after I replaced the function in the SQL with the similar words.
And my question how to deal with the functions?

I have created functions on postgresql and they have appeared but couldn't be uesd. (I got 'No function matches the given name and argument types' after SELECT REGEXP_EXTRACT('test it grabs the number 123', '[0-9]+');)
When I copied the sql and tried to create ventilation view it said 'No function matches the given name and argument types', but it was successfully created after I replaced the function in the SQL with the similar words.
And my question how to deal with the functions?

Functions are created on a specific schema. So it might be that the functions are created on the listed schema, but when you try to use the function it is no longer in the search path. I'm not sure what program you are using but I configure my search path by running set search_path to mimiciii,public;.

OK, I managed to write a .bat file to generate the concepts on Windows. It requires WSL but is probably the most hands-free way of generating the concepts. It works for all but the CCS diagnoses. There's a tutorial in the README: https://github.com/MIT-LCP/mimic-code/tree/master/concepts

Let me know if it works for you all!

OK, I managed to write a .bat file to generate the concepts on Windows. It requires WSL but is probably the most hands-free way of generating the concepts. It works for all but the CCS diagnoses. There's a tutorial in the README: https://github.com/MIT-LCP/mimic-code/tree/master/concepts

Let me know if it works for you all!

Thank you for your guidance! Now I have created functions by using 'set search_path to mimiciii,public' and successfully created some view until

OK, I managed to write a .bat file to generate the concepts on Windows. It requires WSL but is probably the most hands-free way of generating the concepts. It works for all but the CCS diagnoses. There's a tutorial in the README: https://github.com/MIT-LCP/mimic-code/tree/master/concepts

Let me know if it works for you all!

Thank you for your guidance! I changed the path of functions and then it can be uesd. After that, I built views manually following the order from make_concepts.sh and most of it succeed. However here are two problems I met.

1. When I tried to build 'weight_durations', it failed and said 'failed to find convertion from unkonw to text', which affected the creation of ‘heightweight’,'kdigo_uo' and some other views.
2. When I tried to build 'ccs_dx',it failed and said 'The relationship "mimiciii.ccs_multi_dx" does not exist'. Should I create this table first?
   Except these two parts,all views have been built successfully.

1. When I tried to build 'weight_durations', it failed and said 'failed to find convertion from unkonw to text', which affected the creation of ‘heightweight’,'kdigo_uo' and some other views.

That's odd - I built weight_durations perfectly fine. What version of PostgreSQL are you using?

1. When I tried to build 'ccs_dx',it failed and said 'The relationship "mimiciii.ccs_multi_dx" does not exist'. Should I create this table first?

Yes - that I expected. It tries to uncompress the .gz file with gzip, but Windows doesn't come with gzip by default so the building of this table fails. It just requires manually decompressing this table and importing them into the table with a \COPY command.

Except these two parts,all views have been built successfully.

Great!

1. When I tried to build 'weight_durations', it failed and said 'failed to find convertion from unkonw to text', which affected the creation of ‘heightweight’,'kdigo_uo' and some other views.

That's odd - I built weight_durations perfectly fine. What version of PostgreSQL are you using?

I uesd PostgreSQL 9.6 and pgAdmin4. The view has been successfully created after I change LINE69: 'echo' AS weight_type, to 'echo'::text AS weight_type,. At first, I thought it was problem with my view 'echo_data', since it was built automaticly before I tried to create view mannually. And when I try to use the latest code to built another new one to find wether they are different, the new one I created showes 'null' in columns of height, weight, bsa and so on. So I keep the first one.

1. When I tried to build 'ccs_dx',it failed and said 'The relationship "mimiciii.ccs_multi_dx" does not exist'. Should I create this table first?

Yes - that I expected. It tries to uncompress the .gz file with gzip, but Windows doesn't come with gzip by default so the building of this table fails. It just requires manually decompressing this table and importing them into the table with a \COPY command.

I had created the table ccs_multi_dx, and then built view ccs_dx, but it said 'Error: Error: invalid regular expression: parentheses () are not balanced Context: The PL/pgSQL function regexp_contains(text, text) on line 3 of RETURN'?

Except these two parts,all views have been built successfully.

Great!

Dear alistaire:
Thank you all for your help! I noticed that there is a “make-concepts.sql“ file in the mimic-code v1.4.2. I ran it and it worked well.
Can you tell me is there any difference between the two "make-concepts.sh" and "make-concepts.sql" files? I attached the .sql file with this message.
make-concepts.zip
I am totally an outsider for the sql. coding and sorry for all the bothering.
Thanks again.

Can you tell me is there any difference between the two "make-concepts.sh" and "make-concepts.sql" files? I attached the .sql file with this message.

a xxx.sh file is for unix-like and MacOS systems and won't run under Windows. a xxx.sql file is a PostgreSQL script and works as long as you run it in PostgreSQL, the system you use doesn't matter here.