Stoqs: Add "Contour Plot" radio buttons

Full Stack developer skills are needed to implement this new feature. On the client-side there is HTML and JavaScript (JQuery) to be written. AJAX must be understood for passing data between the UI and the server. On the server-side there is Python code that needs to be written to do the plotting. To wrap it all up a functional test will be written using Selenium. (Ideally, to follow true TDD methodology the test will be written first!)

Welcome @J0S349, @noemicuin, @LeslyGJ, @samuelrey! I'm glad you picked this issue. It's something that's been requested by one of our scientists and I think they'll be happy to have this feature.

During the tour of MBARI yesterday we passed by a poster that had an example of the kind of plot that this new feature would provide:

The top plot shows Temperature as colors and also as white contour lines. The bottom plot shows Chlorophyll as color and the white contour lines are the same as in the top plot, Temperature.

The feature to be added to the STOQS UI is the ability to add contour lines of any Parameter.

The test_contour_plots() test added by https://github.com/stoqs/stoqs/pull/501 may be executed from ~/dev/stoqsgit with:

export DJANGO_SETTINGS_MODULE=config.settings.ci
export DATABASE_URL=postgis://127.0.0.1:5432/stoqs
stoqs/manage.py test stoqs.tests.functional_tests.BrowserTestCase.test_contour_plots

If you put back in the debug break point that was removed by https://github.com/stoqs/stoqs/pull/501/commits/b976d708e13fd208df3f29c6f12bd1692e427e38 you should see a window like this:

Note that to execute the functional tests the development server needs to be running. Make sure that this has been executed in another terminal session:

cd ~/dev/stoqsgit && source venv-stoqs/bin/activate
export DATABASE_URL=postgis://stoqsadm:[email protected]:5432/stoqs
stoqs/manage.py runserver 0.0.0.0:8000 --settings=config.settings.ci

The beginnings of a functional test for the "contour lines plot" feature might look like this:

-        # TODO: Add tests for contour line plot
+        # Tests for contour line plot
+        parameter_contour_plot_radio_button = self.browser.find_element(By.XPATH,
+            "//input[@name='parameters_contour_plot' and @value='{}']".format(northward_sea_water_velocity_HR_id))

Which will fail with:

NoSuchElementException: Message: Unable to locate element: //input[@name='parameters_contour_plot' and @value='17']

I was able to add the breakpoint and add the line requesting an unavailable element.

Hi @samuelrey,

That's great! Can you push your code change up to a new branch so that we can all see it?

Hey @MBARIMike I'm having troubles coming up with more test cases.
The one that I have implemented expects an image to be produced when one of the contour radio buttons is pressed.
Another that I thought of was to verify that an equal number of contour buttons as color buttons are produced. This follows my understanding that the radio buttons are dynamically produced according to the parameters described by the data.

Hello @samuelrey (and team),

If you are to follow the example of TDD from Harry Percival's tutorial then your next step is to add code to make the test from https://github.com/stoqs/stoqs/issues/230#issuecomment-280171422 pass.

The test code fails on looking for an input element named parameters_contour_plot. The existing web page has a column of radio buttons named parameters_plot. I suggest looking for this string in file https://github.com/stoqs/stoqs/blob/master/stoqs/stoqs/templates/stoqs/stoqsquery.html and adding similar code to add a column of radio buttons named parameters_contour_plot.

When your test passes commit the changes to your branch. Then write the next test.

Hello @MBARIMike,

We have now implemented all the non-functional radio buttons named as follows: parameters_contour_plot for the contour plot along with the tests to ensure they are generated. Next steps for us will be to implement functionality to the contour radio buttons.

@MBARIMike also we are not available to meet on Friday, March 31 (Cesar Chavez Day) but we would like to meet any other Friday you are available to show you our progress and discuss any suggestions you might have for us. Let us know what works best for you.

Looks like great progress!

Can you provide a link to the branch for these code changes?

Here are the code changes:
https://github.com/samuelrey/stoqs/commit/01fd692e5010fb6aa287899b5d7afb5bd10e9825
https://github.com/samuelrey/stoqs/commit/a3173e7368528c17f3a57ab5daf4ae4aa730153a (at end of file)
https://github.com/samuelrey/stoqs/commit/3122a4ee412b859e949549ebf7e384a61c9fb438 (tests)

In my VM I noticed that the Firefox window appears off screen. I don't know if you encountered that problem, but this change to stoqs/stoqs/tests/functional_tests.py fixes that:

--- a/stoqs/stoqs/tests/functional_tests.py
+++ b/stoqs/stoqs/tests/functional_tests.py
@@ -48,9 +48,6 @@ class BrowserTestCase(TestCase):
     def setUp(self):
         profile = webdriver.FirefoxProfile()
         self.browser = webdriver.Firefox(profile)
-        self.browser.set_window_size(1200, 768)
-        self.browser.set_window_position(300, 0)
-        self.browser.implicitly_wait(10)

Here's a bit of guidance:

1. A little clean up is needed in the column alignment in the title and clear selection rows
2. Add a handler for clicks on the parameters_contour_plot named radio buttons. This JavaScript code is the handler for the existing Plot buttons. It builds a querystring that is then passed to the server where the contour plot will be generated. Similar code needs to be added to add the parameters_contour_plot to the querystring.

The next steps start getting a little gnarly, as they get into the server-side code:

1. Add lines for parametercontourplot to the stoqs/stoqs/views/query.py Python code
2. In file stoqs/utils/STOQSQManager.py around here add code to pull the contourparameterID and contourparameterGroups and add them to the argument list for MeasuredParameter().
3. In file stoqs/utils/Viz/plotting.py add contourparameterID and contourparameterGroups to MeasuredParameter()'s __init__()

At this point your functional test stoqs.tests.functional_tests.BrowserTestCase.test_contour_plots should pass and you will have access to the parameter ID that has been selected for contouring where you need it (the renderDatavaluesForFlot() method of the MeasuredParameter class).

I'll be taking up the javascript function!

Last commit in all of its raw glory. We have finished everything listed in this comment. Now we are working on actually plotting the data! Modifications will be made to plotting.py.

I ran into a problem when trying to generate color plots. data['parameterplatformdatavaluepng'] has [null, null, 'Problem with getting parameter-contour-plot-radio button info']. Set a breakpoint here, select any of the color plot radio buttons & verify that the data is the same as I've observed. Still narrowing down the cause.

The 'Problem with getting parameter-contour-plot-radio button info' text is coming from this line.

Oh, that's my wrong doing.
My guess for this error is that 'platformName' should be 'contourplatformName'? Or another solution may be to not check for 'contourplatformName'?
Any advice would help to avoid an unneeded amount of commits.

The _fillXYZ & loadData methods have a lot of code that can be reused for the contour lines variables. Maybe instead of setting the member variables within the function, we can modify the functions to return the lists that they generate.

if contourParameterID:
    self.clx, self.cly, self.clz = self.loadData(...)

Thinking about how best to reuse the loadData() method in plotting.py to fill data for the contour line x, y, and z variables. What do you think about a modification like this near here?

diff --git a/stoqs/utils/Viz/plotting.py b/stoqs/utils/Viz/plotting.py
index 81c278b..cfe679c 100644
--- a/stoqs/utils/Viz/plotting.py
+++ b/stoqs/utils/Viz/plotting.py
@@ -556,6 +556,12 @@ class MeasuredParameter(BaseParameter):

             if not self.x and not self.y and not self.z:
                 self.loadData()
+            if contourParameterID is not None:
+                if not self.clx and not self.cly and not self.clz:
+                    self.loadData()
+                    self.clx = self.x
+                    self.cly = self.y
+                    self.clz = self.z

Please see this Pull Request that dives deep into the "rabbit hole" to produce parallel contour_qs_mp and help you reuse loadData() to get the contour line parameter data values. You should now be able to use the clx, cly, and clz variables to make the contour lines.

@samuelrey Please change the base branch of to https://github.com/stoqs/stoqs/pull/578 to the capstone-spring2017 branch. Here are instructions on how to do that: https://github.com/blog/2224-change-the-base-branch-of-a-pull-request

With it being on a separate branch it will be easier to manage incremental merges.

I'll get it done by the end of the day! It's good to go.

Looks good! Did you also see this small request?

Right before the figure is saved, I insert

                if self.contourParameterID is not None:
                    CS = ax.contour(clx, cly, clz)
                    ax.clabel(CS, fontsize=9, inline=1)

which throws an error regarding the shape of z.
The contour documentation says that x, y and z must either be 2d with the same shape or x and y must be 1d with length equal to the number of rows in z.

You need to pass in gridded variables, produced in the same way that xi, yi, and zi are produced.

Oh I over looked that. Got it!

Here's a look at what our code was able to produce. The UI and image quality will look different now because of updates made by Mike; however, this is what the CSUMB STOQS team was able to deliver.

To provide a quick summary of the work the CSUMB STOQS team completed, we filtered through the already existing code and made adjustments as necessary. One of these adjustments was adding a whole new column of radio buttons so the UI had the option for mapping contour lines; these UI changes were made by myself and @LeslyGJ. Following the new format, connections to those buttons had to be made as well; making changes to the JavaScript @samuelrey took a hold of that task. Another adjustment came when diving into the python behind the server-side code; @samuelrey and myself added new variables and passed through new parameters. With UI changes, and server-side modifications, @J0S349 and @samuelrey monitored test completions by creating new test cases or making changes wherever needed to get the code passing and back on track.