Plots2: Remove "-" before count of posts on Topic Cards when sorting by "# of People"

Created on 28 May 2020  ·  25Comments  ·  Source: publiclab/plots2

Please describe the problem

There appears to be a tiny "-" before the count of posts on Topic Cards when the "# of People" sorting option is clicked, making it appear like there is a negative count of posts. See screenshot:

Screen Shot 2020-05-20 at 10 36 42 AM (1)

What did you expect to see that you didn't?

i expected to see positive integers.

Please show us where to look

https://publiclab.org/tags?sort=followers&order=asc

What's your PublicLab.org username?

This can help us diagnose the issue:

liz

bug

Most helpful comment

Hey @ebarry I think this is a more serious issue actually :sweat_smile: I had made changes for this here https://github.com/publiclab/plots2/pull/7609 but I must have messed it up and calculated the wrong count I think :disappointed: Looping in @jywarren Sorry might need some help to fix this up :sweat_smile:

All 25 comments

Does anyone have code links at hand that they could add here? Thanks in advance!

Hey @ebarry I think this is a more serious issue actually :sweat_smile: I had made changes for this here https://github.com/publiclab/plots2/pull/7609 but I must have messed it up and calculated the wrong count I think :disappointed: Looping in @jywarren Sorry might need some help to fix this up :sweat_smile:

Hi @Tlazypanda can we look at other places where Tag.counter type code is used, and try to switch to that?

https://github.com/publiclab/plots2/pull/7609/files#diff-8fe92870a7f027a46eed82a51c5c16bcR118

However, the counts are off by very little. I wonder if this is an issue of some things being spammed and then that affecting the math a little bit?

See for example the difference between these counts:

image

image

We can look for where the 236 count is calculated and switch over to that, perhaps?

So that uses a controller-generated instance variable:

https://github.com/publiclab/plots2/blob/c056cb0b97ebc6b34184ec3dfa9a83e1096395f9/app/views/tag/show.html.erb#L58

I believe this is where this is calculated:

https://github.com/publiclab/plots2/blob/c056cb0b97ebc6b34184ec3dfa9a83e1096395f9/app/controllers/tag_controller.rb#L551-L560

Hey @jywarren, what I can see in tag/:id i.e. the showpage for tag is that we are displaying only research notes over there and in topic cards , we are trying to display research notes+questions.
So, I think what we need to do is that we need to fetch notes in the same way as we are doing in tag#show and for displaying the count at the bottom of topic card we will just need to use this math expression:
Number of notes with the tagname-Count of displayed notes in Topic Card

we need to fetch notes in the same way as we are doing in tag#show

Hi, that's just right! noting the concept of a "single source of truth" here: https://github.com/publiclab/plots2/issues/6855#issuecomment-578331475

I believe we should be using the same calculations here: https://github.com/publiclab/plots2/pull/7476 (based on #6855).

@urvashigupta7 thank you so much for chiming in! Would you be interested in making these changes?

I also think that in https://github.com/publiclab/plots2/pull/7476/, we are not filtering out spam. So on these lines, we should probably add .where(status: 1):

https://github.com/publiclab/plots2/blob/c056cb0b97ebc6b34184ec3dfa9a83e1096395f9/app/controllers/tag_controller.rb#L551-L560

That code also depends on this, which doesn't filter spam:

https://github.com/publiclab/plots2/blob/f4dfdcbae4dfa60b0ec357081ee1fae6f2089436/app/models/node.rb#L609-L618

Likewise this code needs also to filter spam out with the same filter:

https://github.com/publiclab/plots2/blob/f4dfdcbae4dfa60b0ec357081ee1fae6f2089436/app/models/tag.rb#L118-L124

Hmm. It does seem like we are filtering spam from the @total_posts calculation, as it runs /after/ these lines:

https://github.com/publiclab/plots2/blob/c056cb0b97ebc6b34184ec3dfa9a83e1096395f9/app/controllers/tag_controller.rb#L204-L218

However, neither Tag.counter and Node.for_tagname_and_type filter. Maybe that is the irregularity?

The count here is still not right, i think. For example https://stable.publiclab.org/tags now lists 692 posts for balloon-mapping but here are the stats for that tag (in stable):

image

I think we need to depend on this fix: https://github.com/publiclab/plots2/pull/8245

then we can check this again. Reopening for a moment... fingers 🤞

https://github.com/publiclab/plots2/pull/8245 is now done! And has unit tests, so nice. OK, now this may work...

OK, it says 573 posts » -- hmm

Research notes 469
Questions 37
Wiki pages 77
People 346

I mean, notes plus wikis = 546...? and plus questions is 583?

Now, testing the tag tests on stable, I'm suspicious that there may be some caching going on. Adding that tag to a new note, https://stable.publiclab.org/notes/aidanswope/05-21-2019/angel-s-point-balloon-map-updated, did not result in the tab of notes going from 7 to 8, although tag.count returns 8 as does the result of tag.run_count

https://stable.publiclab.org/tag/tests

OK, so tracing the tab count:

https://github.com/publiclab/plots2/blob/e97eff7464a544a97f4b38f169f76250600b3148/app/models/node.rb#L621-L630

Node.where(status: 1, type: 'note').includes(:revision, :tag).references(:term_data, :node_revisions).where('term_data.name = ? OR term_data.parent = ?', 'balloon-mapping', 'balloon-mapping').where.not(nid: qids).count => 470

Versus the topic card count:

NodeTag.joins(:node).where(tid: 10).where('node.status = 1 AND node.type = "note"').where.not(nid: qids).count => 467

😭

These three are the difference: [6129, 7236, 7414] --

[#,

,

]

VERY strange! for 2 reasons: 3 notes is not the same as the 10 missing ones (thought we haven't looked at questions yet), AND, these three seem to be normal except that one was in Feb 2013 and 2 were in May 2013:

(update: see below for duplicate tagnames)

Also interesting:

Node.where(status: 1, type: 'note').includes(:revision, :tag).references(:term_data, :node_revisions).where('term_data.name = ? OR term_data.parent = ?', 'balloon-mapping', 'balloon-mapping').where(nid: qids).count => 29

This agrees with:

NodeTag.joins(:node).where(tid: 10).where('node.status = 1 AND node.type = "note"').where(nid: qids).count

vs. 37 as shown on the tab, which is calculated with:

https://github.com/publiclab/plots2/blob/e97eff7464a544a97f4b38f169f76250600b3148/app/models/node.rb#L640-L646

Aha! That last section shows that posts tagged question:balloon-mapping are included in that count even if they do not have the base balloon-mapping tag!!!

The following 8 posts seem to have question:balloon-mapping but not balloon-mapping:

[13556, 15762, 16426, 18994, 20694, 11301, 12251, 19625]

OK, so that gets us to within 2 nodes. Now let's just manually compare:

irb(main):040:0> NodeTag.joins(:node).where(tid: 10).where('node.status = 1 AND node.type = "note"').count
=> 496
irb(main):041:0> Node.where(status: 1, type: 'note').includes(:tag).references(:term_data).where('term_data.name = ?', 'balloon-mapping').count
=> 499
irb(main):022:0> Node.where(status: 1, type: 'note').includes(:revision, :tag).references(:term_data, :node_revisions).where('term_data.name = ? OR term_data.parent = ?', 'balloon-mapping', 'balloon-mapping').collect(&:nid)
=> [146, 253, 254, 268, 403, 511, 724, 732, 743, 913, 1482, 1489, 1620, 1626, 1627, 1632, 1715, 1849, 1968, 1994, 2126, 2138, 2144, 2476, 2589, 2930, 3115, 3325, 4649, 4952, 5038, 5220, 5250, 5978, 6934, 7050, 7189, 7294, 7314, 7491, 7577, 7593, 7995, 8189, 8907, 8913, 8914, 8950, 9019, 9032, 9042, 9086, 9417, 9434, 9742, 9888, 9977, 10541, 11019, 11100, 11223, 11574, 11582, 11827, 12201, 12240, 12491, 12619, 12708, 13095, 13424, 13945, 14022, 14095, 14183, 14185, 14219, 14515, 14572, 14587, 14597, 14600, 14611, 14620, 14813, 14907, 14909, 14983, 15545, 15568, 16591, 16653, 16746, 16877, 18316, 19153, 19272, 19462, 19558, 19578, 19620, 19762, 20331, 23549, 24101, 24102, 1795, 6635, 347, 369, 439, 444, 792, 861, 1665, 1668, 1962, 2135, 2143, 2152, 2245, 2277, 2783, 2784, 3563, 3635, 3670, 4482, 4516, 4755, 4760, 4923, 4925, 6059, 7641, 10181, 10489, 11101, 11724, 13243, 13880, 13948, 17322, 64, 166, 202, 344, 1666, 1816, 1818, 1819, 2919, 3354, 3736, 3821, 4087, 8020, 8947, 9301, 11199, 11260, 11261, 11755, 12161, 1544, 1739, 1742, 1892, 2006, 5494, 9353, 10705, 828, 1515, 1786, 1877, 1926, 1949, 2121, 2625, 2666, 2677, 2846, 2892, 2961, 4092, 4907, 5394, 5415, 5664, 7132, 7367, 7368, 7717, 7964, 8400, 10243, 6175, 9507, 4636, 6293, 3014, 6624, 9806, 10289, 10554, 11278, 11391, 12104, 12622, 12894, 13143, 9935, 10036, 10196, 10450, 11283, 11719, 11769, 12237, 10246, 10605, 2462, 9497, 1889, 2131, 2443, 3026, 4127, 4252, 4673, 5538, 5893, 6608, 8034, 8902, 8959, 9644, 9707, 10045, 10162, 10335, 10336, 10337, 10373, 10540, 10570, 11107, 11110, 11112, 11124, 11143, 11175, 11298, 11336, 12053, 12203, 12590, 16984, 19515, 19580, 23822, 2163, 3634, 7620, 10166, 11056, 11319, 11589, 1541, 1652, 1676, 1677, 1746, 2154, 2237, 2275, 2311, 2623, 2785, 3454, 3652, 3822, 4014, 4691, 4720, 4950, 5161, 7785, 8866, 8926, 8928, 8937, 8958, 8986, 9759, 9915, 10101, 10230, 10244, 10302, 10311, 10374, 10523, 10602, 11021, 11078, 11103, 11253, 11726, 11988, 12686, 12752, 12757, 13400, 13845, 14116, 14660, 14713, 14756, 1716, 2660, 7191, 7192, 1743, 1636, 1900, 9317, 10651, 1741, 13904, 3096, 2352, 2824, 4091, 2715, 6080, 6082, 2732, 3801, 3908, 4545, 7121, 10284, 13791, 10631, 4921, 6751, 9429, 9743, 11577, 17331, 6073, 9858, 10711, 11118, 6525, 10267, 9034, 11252, 11268, 8033, 8868, 10159, 10214, 9027, 9785, 9033, 9076, 9299, 14501, 14508, 14562, 21733, 10282, 9909, 11178, 11738, 10024, 12667, 11262, 11343, 11978, 11983, 11992, 11826, 11061, 10227, 10428, 11077, 11131, 11137, 11280, 11281, 11518, 11551, 11807, 11812, 11947, 12039, 14158, 14729, 14943, 15641, 16273, 16349, 18560, 19269, 19286, 21547, 10343, 14852, 14901, 15352, 15387, 10612, 10638, 10986, 10994, 20181, 11002, 11127, 11128, 11183, 12166, 11342, 11494, 11955, 11673, 11707, 11711, 12190, 18839, 13802, 14595, 12227, 15912, 14007, 13093, 13096, 12981, 13102, 13160, 13194, 13196, 13227, 13242, 13267, 13278, 13281, 13298, 14168, 15161, 23143, 14058, 14766, 13748, 13839, 15603, 14241, 14179, 14239, 14246, 14240, 14255, 14488, 14487, 14245, 14785, 15160, 14703, 14995, 16571, 15172, 15179, 15424, 15713, 15732, 16141, 16284, 16334, 16382, 17387, 17498, 17499, 18498, 16960, 19689, 17815, 19394, 19464, 19650, 19955, 19476, 19477, 23203, 6129, 7236, 7414]

vs:


irb(main):024:0> NodeTag.joins(:node).where(tid: 10).where('node.status = 1 AND node.type = "note"').collect(&:nid)
=> [146, 253, 254, 268, 403, 511, 724, 732, 743, 913, 1482, 1489, 1620, 1626, 1627, 1632, 1715, 1849, 1968, 1994, 2126, 2138, 2144, 2476, 2589, 2930, 3115, 3325, 4649, 4952, 5038, 5220, 5250, 5978, 6934, 7050, 7189, 7294, 7314, 7491, 7577, 7593, 7995, 8189, 8907, 8913, 8914, 8950, 9019, 9032, 9042, 9086, 9417, 9434, 9742, 9888, 9977, 10541, 11019, 11100, 11223, 11574, 11582, 11827, 12201, 12240, 12491, 12619, 12708, 13095, 13424, 13945, 14022, 14095, 14183, 14185, 14219, 14515, 14572, 14587, 14597, 14600, 14611, 14620, 14813, 14907, 14909, 14983, 15545, 15568, 16591, 16653, 16746, 16877, 18316, 19153, 19272, 19462, 19558, 19578, 19620, 19762, 20331, 23549, 24101, 24102, 1795, 6635, 347, 369, 439, 444, 792, 861, 1665, 1668, 1962, 2135, 2143, 2152, 2245, 2277, 2783, 2784, 3563, 3635, 3670, 4482, 4516, 4755, 4760, 4923, 4925, 6059, 7641, 10181, 10489, 11101, 11724, 13243, 13880, 13948, 17322, 64, 166, 202, 344, 1666, 1816, 1818, 1819, 2919, 3354, 3736, 3821, 4087, 8020, 8947, 9301, 11199, 11260, 11261, 11755, 12161, 1544, 1739, 1742, 1892, 2006, 5494, 9353, 10705, 828, 1515, 1786, 1877, 1926, 1949, 2121, 2625, 2666, 2677, 2846, 2892, 2961, 4092, 4907, 5394, 5415, 5664, 7132, 7367, 7368, 7717, 7964, 8400, 10243, 6175, 9507, 4636, 6293, 3014, 6624, 9806, 10289, 10554, 11278, 11391, 12104, 12622, 12894, 13143, 9935, 10036, 10196, 10450, 11283, 11719, 11769, 12237, 10246, 10605, 2462, 9497, 1889, 2131, 2443, 3026, 4127, 4252, 4673, 5538, 5893, 6608, 8034, 8902, 8959, 9644, 9707, 10045, 10162, 10335, 10336, 10337, 10373, 10540, 10570, 11107, 11110, 11112, 11124, 11143, 11175, 11298, 11336, 12053, 12203, 12590, 16984, 19515, 19580, 23822, 2163, 3634, 7620, 10166, 11056, 11319, 11589, 1541, 1652, 1676, 1677, 1746, 2154, 2237, 2275, 2311, 2623, 2785, 3454, 3652, 3822, 4014, 4691, 4720, 4950, 5161, 7785, 8866, 8926, 8928, 8937, 8958, 8986, 9759, 9915, 10101, 10230, 10244, 10302, 10311, 10374, 10523, 10602, 11021, 11078, 11103, 11253, 11726, 11988, 12686, 12752, 12757, 13400, 13845, 14116, 14660, 14713, 14756, 1716, 2660, 7191, 7192, 1743, 1636, 1900, 9317, 10651, 1741, 13904, 3096, 2352, 2824, 4091, 2715, 6080, 6082, 2732, 3801, 3908, 4545, 7121, 10284, 13791, 10631, 4921, 6751, 9429, 9743, 11577, 17331, 6073, 9858, 10711, 11118, 6525, 10267, 9034, 11252, 11268, 8033, 8868, 10159, 10214, 9027, 9785, 9033, 9076, 9299, 14501, 14508, 14562, 21733, 10282, 9909, 11178, 11738, 10024, 12667, 11262, 11343, 11978, 11983, 11992, 11826, 11061, 10227, 10428, 11077, 11131, 11137, 11280, 11281, 11518, 11551, 11807, 11812, 11947, 12039, 14158, 14729, 14943, 15641, 16273, 16349, 18560, 19269, 19286, 21547, 10343, 14852, 14901, 15352, 15387, 10612, 10638, 10986, 10994, 20181, 11002, 11127, 11128, 11183, 12166, 11342, 11494, 11955, 11673, 11707, 11711, 12190, 18839, 13802, 14595, 12227, 15912, 14007, 13093, 13096, 12981, 13102, 13160, 13194, 13196, 13227, 13242, 13267, 13278, 13281, 13298, 14168, 15161, 23143, 14058, 14766, 13748, 13839, 15603, 14241, 14179, 14239, 14246, 14240, 14255, 14488, 14487, 14245, 14785, 15160, 14703, 14995, 16571, 15172, 15179, 15424, 15713, 15732, 16141, 16284, 16334, 16382, 17387, 17498, 17499, 18498, 16960, 19689, 17815, 19394, 19464, 19650, 19955, 19476, 19477, 23203]

Aha. Maybe this:

Tag.where(name: 'balloon-mapping').length => 4
Tag.where(name: 'balloon-mapping').collect(&:tid) => [10, 3205, 3591, 4032]

Yes, that's the discrepancy:

irb(main):045:0> NodeTag.joins(:node).where(tid: [10, 3205, 3591, 4032]).where('node.status = 1 AND node.type = "note"').count
=> 499
irb(main):046:0> Node.where(status: 1, type: 'note').includes(:tag).references(:term_data).where('term_data.name = ?', 'balloon-mapping').count
=> 499

irb(main):050:0> Tag.where(name: 'balloon-mapping').collect(&:count)
=> [574, 1, 1, 1]

However, notably not for the method or project bugs in #7334

irb(main):047:0> Tag.where(name: 'method').collect(&:tid)
=> [14474]
irb(main):048:0> Tag.where(name: 'project').collect(&:tid)
=> [14751]

OK, i fixed the duplicate tags with:

irb(main):053:0> NodeTag.where(tid: [3205, 3591, 4032]).each do |nt|
irb(main):054:1* nt.tid = 10
irb(main):055:1> nt.save
irb(main):056:1> end

however i bet there are more in the DB. We may want to do a bigger project to consolidate these.

Update: yes, there are 28 other instances:

irb(main):063:0> Tag.count
=> 18532
irb(main):065:0> Tag.all.collect(&:name).uniq.count
=> 18504


irb(main):069:0> (Tag.all - Tag.all.group(:name)).collect(&:name)
=> ["fold-up", "gulf-coast", "kite-mapping", "gulf-coast", "brooklyn", "new-york-city", "balloon-mapping", "spectrometer", "farmhack", "white-balance", "fresh-kills", "balloon-mapping", "balloon-mapping", "kite-mapping", "newsletter", "newsletter", "passenger-pigeon", "jamaica-bay", "mobile-spectrometer", "infragram-sandbox", "mailing-lists", "diy-kite", "water-sensing", "carbon-arc", "diy-kite", "diy-kite", "workplace", "emergencies"]

My recommendation now is:

We've narrowed this down a lot, and it has two parts.

  1. “ghost” tags - duplicate tagnames so basically some weren’t being counted (solved, although ~15 other instances exist for a few other tags)
  2. question:_____ format tags, which are counted under the questions tab on /tag/____ pages, but if they don’t have their own base tag, are not counted in the topic cards. Example: 8 posts have question:balloon-mapping but not balloon-mapping , so they throw our count by 8.

My take on (2) is that we should write it down and be aware of it, but not solve it (i can’t think of a clean solution anyways).

And, even after all that digging, I still have a discrepancy of 1 for balloon-mapping. But I think we are so close that we should essentially be OK with that, as it’s one of the largest tags and that issue is unlikely to affect others. Interested to hear others' take on that too!

Was this page helpful?
0 / 5 - 0 ratings