There appears to be a tiny "-" before the count of posts on Topic Cards when the "# of People" sorting option is clicked, making it appear like there is a negative count of posts. See screenshot:
What did you expect to see that you didn't?
i expected to see positive integers.
https://publiclab.org/tags?sort=followers&order=asc
This can help us diagnose the issue:
liz
Does anyone have code links at hand that they could add here? Thanks in advance!
Hey @ebarry I think this is a more serious issue actually :sweat_smile: I had made changes for this here https://github.com/publiclab/plots2/pull/7609 but I must have messed it up and calculated the wrong count I think :disappointed: Looping in @jywarren Sorry might need some help to fix this up :sweat_smile:
Hi @Tlazypanda can we look at other places where Tag.counter
type code is used, and try to switch to that?
https://github.com/publiclab/plots2/pull/7609/files#diff-8fe92870a7f027a46eed82a51c5c16bcR118
However, the counts are off by very little. I wonder if this is an issue of some things being spammed and then that affecting the math a little bit?
See for example the difference between these counts:
We can look for where the 236 count is calculated and switch over to that, perhaps?
So that uses a controller-generated instance variable:
I believe this is where this is calculated:
Hey @jywarren, what I can see in tag/:id i.e. the showpage for tag is that we are displaying only research notes over there and in topic cards , we are trying to display research notes+questions.
So, I think what we need to do is that we need to fetch notes in the same way as we are doing in tag#show and for displaying the count at the bottom of topic card we will just need to use this math expression:
Number of notes with the tagname-Count of displayed notes in Topic Card
we need to fetch notes in the same way as we are doing in tag#show
Hi, that's just right! noting the concept of a "single source of truth" here: https://github.com/publiclab/plots2/issues/6855#issuecomment-578331475
I believe we should be using the same calculations here: https://github.com/publiclab/plots2/pull/7476 (based on #6855).
@urvashigupta7 thank you so much for chiming in! Would you be interested in making these changes?
I also think that in https://github.com/publiclab/plots2/pull/7476/, we are not filtering out spam. So on these lines, we should probably add .where(status: 1)
:
That code also depends on this, which doesn't filter spam:
Likewise this code needs also to filter spam out with the same filter:
Hmm. It does seem like we are filtering spam from the @total_posts
calculation, as it runs /after/ these lines:
However, neither Tag.counter
and Node.for_tagname_and_type
filter. Maybe that is the irregularity?
The count here is still not right, i think. For example https://stable.publiclab.org/tags now lists 692 posts for balloon-mapping
but here are the stats for that tag (in stable):
I think we need to depend on this fix: https://github.com/publiclab/plots2/pull/8245
then we can check this again. Reopening for a moment... fingers 🤞
https://github.com/publiclab/plots2/pull/8245 is now done! And has unit tests, so nice. OK, now this may work...
OK, it says 573 posts »
-- hmm
Research notes 469
Questions 37
Wiki pages 77
People 346
I mean, notes plus wikis = 546...? and plus questions is 583?
Now, testing the tag tests
on stable, I'm suspicious that there may be some caching going on. Adding that tag to a new note, https://stable.publiclab.org/notes/aidanswope/05-21-2019/angel-s-point-balloon-map-updated, did not result in the tab of notes going from 7 to 8, although tag.count
returns 8 as does the result of tag.run_count
and noting link to https://github.com/publiclab/plots2/issues/8244
YESSSSS it's 5-minute cached: https://github.com/publiclab/plots2/blob/e97eff7464a544a97f4b38f169f76250600b3148/app/views/tag/show.html.erb#L133
OK, so tracing the tab count:
Node.where(status: 1, type: 'note').includes(:revision, :tag).references(:term_data, :node_revisions).where('term_data.name = ? OR term_data.parent = ?', 'balloon-mapping', 'balloon-mapping').where.not(nid: qids).count
=> 470
Versus the topic card count:
NodeTag.joins(:node).where(tid: 10).where('node.status = 1 AND node.type = "note"').where.not(nid: qids).count
=> 467
😭
These three are the difference: [6129, 7236, 7414]
--
[#
,
,
]
VERY strange! for 2 reasons: 3 notes is not the same as the 10 missing ones (thought we haven't looked at questions yet), AND, these three seem to be normal except that one was in Feb 2013 and 2 were in May 2013:
(update: see below for duplicate tagnames)
Also interesting:
Node.where(status: 1, type: 'note').includes(:revision, :tag).references(:term_data, :node_revisions).where('term_data.name = ? OR term_data.parent = ?', 'balloon-mapping', 'balloon-mapping').where(nid: qids).count
=> 29
This agrees with:
NodeTag.joins(:node).where(tid: 10).where('node.status = 1 AND node.type = "note"').where(nid: qids).count
vs. 37 as shown on the tab, which is calculated with:
Aha! That last section shows that posts tagged question:balloon-mapping
are included in that count even if they do not have the base balloon-mapping
tag!!!
The following 8 posts seem to have question:balloon-mapping
but not balloon-mapping
:
[13556, 15762, 16426, 18994, 20694, 11301, 12251, 19625]
OK, so that gets us to within 2 nodes. Now let's just manually compare:
irb(main):040:0> NodeTag.joins(:node).where(tid: 10).where('node.status = 1 AND node.type = "note"').count
=> 496
irb(main):041:0> Node.where(status: 1, type: 'note').includes(:tag).references(:term_data).where('term_data.name = ?', 'balloon-mapping').count
=> 499
irb(main):022:0> Node.where(status: 1, type: 'note').includes(:revision, :tag).references(:term_data, :node_revisions).where('term_data.name = ? OR term_data.parent = ?', 'balloon-mapping', 'balloon-mapping').collect(&:nid)
=> [146, 253, 254, 268, 403, 511, 724, 732, 743, 913, 1482, 1489, 1620, 1626, 1627, 1632, 1715, 1849, 1968, 1994, 2126, 2138, 2144, 2476, 2589, 2930, 3115, 3325, 4649, 4952, 5038, 5220, 5250, 5978, 6934, 7050, 7189, 7294, 7314, 7491, 7577, 7593, 7995, 8189, 8907, 8913, 8914, 8950, 9019, 9032, 9042, 9086, 9417, 9434, 9742, 9888, 9977, 10541, 11019, 11100, 11223, 11574, 11582, 11827, 12201, 12240, 12491, 12619, 12708, 13095, 13424, 13945, 14022, 14095, 14183, 14185, 14219, 14515, 14572, 14587, 14597, 14600, 14611, 14620, 14813, 14907, 14909, 14983, 15545, 15568, 16591, 16653, 16746, 16877, 18316, 19153, 19272, 19462, 19558, 19578, 19620, 19762, 20331, 23549, 24101, 24102, 1795, 6635, 347, 369, 439, 444, 792, 861, 1665, 1668, 1962, 2135, 2143, 2152, 2245, 2277, 2783, 2784, 3563, 3635, 3670, 4482, 4516, 4755, 4760, 4923, 4925, 6059, 7641, 10181, 10489, 11101, 11724, 13243, 13880, 13948, 17322, 64, 166, 202, 344, 1666, 1816, 1818, 1819, 2919, 3354, 3736, 3821, 4087, 8020, 8947, 9301, 11199, 11260, 11261, 11755, 12161, 1544, 1739, 1742, 1892, 2006, 5494, 9353, 10705, 828, 1515, 1786, 1877, 1926, 1949, 2121, 2625, 2666, 2677, 2846, 2892, 2961, 4092, 4907, 5394, 5415, 5664, 7132, 7367, 7368, 7717, 7964, 8400, 10243, 6175, 9507, 4636, 6293, 3014, 6624, 9806, 10289, 10554, 11278, 11391, 12104, 12622, 12894, 13143, 9935, 10036, 10196, 10450, 11283, 11719, 11769, 12237, 10246, 10605, 2462, 9497, 1889, 2131, 2443, 3026, 4127, 4252, 4673, 5538, 5893, 6608, 8034, 8902, 8959, 9644, 9707, 10045, 10162, 10335, 10336, 10337, 10373, 10540, 10570, 11107, 11110, 11112, 11124, 11143, 11175, 11298, 11336, 12053, 12203, 12590, 16984, 19515, 19580, 23822, 2163, 3634, 7620, 10166, 11056, 11319, 11589, 1541, 1652, 1676, 1677, 1746, 2154, 2237, 2275, 2311, 2623, 2785, 3454, 3652, 3822, 4014, 4691, 4720, 4950, 5161, 7785, 8866, 8926, 8928, 8937, 8958, 8986, 9759, 9915, 10101, 10230, 10244, 10302, 10311, 10374, 10523, 10602, 11021, 11078, 11103, 11253, 11726, 11988, 12686, 12752, 12757, 13400, 13845, 14116, 14660, 14713, 14756, 1716, 2660, 7191, 7192, 1743, 1636, 1900, 9317, 10651, 1741, 13904, 3096, 2352, 2824, 4091, 2715, 6080, 6082, 2732, 3801, 3908, 4545, 7121, 10284, 13791, 10631, 4921, 6751, 9429, 9743, 11577, 17331, 6073, 9858, 10711, 11118, 6525, 10267, 9034, 11252, 11268, 8033, 8868, 10159, 10214, 9027, 9785, 9033, 9076, 9299, 14501, 14508, 14562, 21733, 10282, 9909, 11178, 11738, 10024, 12667, 11262, 11343, 11978, 11983, 11992, 11826, 11061, 10227, 10428, 11077, 11131, 11137, 11280, 11281, 11518, 11551, 11807, 11812, 11947, 12039, 14158, 14729, 14943, 15641, 16273, 16349, 18560, 19269, 19286, 21547, 10343, 14852, 14901, 15352, 15387, 10612, 10638, 10986, 10994, 20181, 11002, 11127, 11128, 11183, 12166, 11342, 11494, 11955, 11673, 11707, 11711, 12190, 18839, 13802, 14595, 12227, 15912, 14007, 13093, 13096, 12981, 13102, 13160, 13194, 13196, 13227, 13242, 13267, 13278, 13281, 13298, 14168, 15161, 23143, 14058, 14766, 13748, 13839, 15603, 14241, 14179, 14239, 14246, 14240, 14255, 14488, 14487, 14245, 14785, 15160, 14703, 14995, 16571, 15172, 15179, 15424, 15713, 15732, 16141, 16284, 16334, 16382, 17387, 17498, 17499, 18498, 16960, 19689, 17815, 19394, 19464, 19650, 19955, 19476, 19477, 23203, 6129, 7236, 7414]
vs:
irb(main):024:0> NodeTag.joins(:node).where(tid: 10).where('node.status = 1 AND node.type = "note"').collect(&:nid)
=> [146, 253, 254, 268, 403, 511, 724, 732, 743, 913, 1482, 1489, 1620, 1626, 1627, 1632, 1715, 1849, 1968, 1994, 2126, 2138, 2144, 2476, 2589, 2930, 3115, 3325, 4649, 4952, 5038, 5220, 5250, 5978, 6934, 7050, 7189, 7294, 7314, 7491, 7577, 7593, 7995, 8189, 8907, 8913, 8914, 8950, 9019, 9032, 9042, 9086, 9417, 9434, 9742, 9888, 9977, 10541, 11019, 11100, 11223, 11574, 11582, 11827, 12201, 12240, 12491, 12619, 12708, 13095, 13424, 13945, 14022, 14095, 14183, 14185, 14219, 14515, 14572, 14587, 14597, 14600, 14611, 14620, 14813, 14907, 14909, 14983, 15545, 15568, 16591, 16653, 16746, 16877, 18316, 19153, 19272, 19462, 19558, 19578, 19620, 19762, 20331, 23549, 24101, 24102, 1795, 6635, 347, 369, 439, 444, 792, 861, 1665, 1668, 1962, 2135, 2143, 2152, 2245, 2277, 2783, 2784, 3563, 3635, 3670, 4482, 4516, 4755, 4760, 4923, 4925, 6059, 7641, 10181, 10489, 11101, 11724, 13243, 13880, 13948, 17322, 64, 166, 202, 344, 1666, 1816, 1818, 1819, 2919, 3354, 3736, 3821, 4087, 8020, 8947, 9301, 11199, 11260, 11261, 11755, 12161, 1544, 1739, 1742, 1892, 2006, 5494, 9353, 10705, 828, 1515, 1786, 1877, 1926, 1949, 2121, 2625, 2666, 2677, 2846, 2892, 2961, 4092, 4907, 5394, 5415, 5664, 7132, 7367, 7368, 7717, 7964, 8400, 10243, 6175, 9507, 4636, 6293, 3014, 6624, 9806, 10289, 10554, 11278, 11391, 12104, 12622, 12894, 13143, 9935, 10036, 10196, 10450, 11283, 11719, 11769, 12237, 10246, 10605, 2462, 9497, 1889, 2131, 2443, 3026, 4127, 4252, 4673, 5538, 5893, 6608, 8034, 8902, 8959, 9644, 9707, 10045, 10162, 10335, 10336, 10337, 10373, 10540, 10570, 11107, 11110, 11112, 11124, 11143, 11175, 11298, 11336, 12053, 12203, 12590, 16984, 19515, 19580, 23822, 2163, 3634, 7620, 10166, 11056, 11319, 11589, 1541, 1652, 1676, 1677, 1746, 2154, 2237, 2275, 2311, 2623, 2785, 3454, 3652, 3822, 4014, 4691, 4720, 4950, 5161, 7785, 8866, 8926, 8928, 8937, 8958, 8986, 9759, 9915, 10101, 10230, 10244, 10302, 10311, 10374, 10523, 10602, 11021, 11078, 11103, 11253, 11726, 11988, 12686, 12752, 12757, 13400, 13845, 14116, 14660, 14713, 14756, 1716, 2660, 7191, 7192, 1743, 1636, 1900, 9317, 10651, 1741, 13904, 3096, 2352, 2824, 4091, 2715, 6080, 6082, 2732, 3801, 3908, 4545, 7121, 10284, 13791, 10631, 4921, 6751, 9429, 9743, 11577, 17331, 6073, 9858, 10711, 11118, 6525, 10267, 9034, 11252, 11268, 8033, 8868, 10159, 10214, 9027, 9785, 9033, 9076, 9299, 14501, 14508, 14562, 21733, 10282, 9909, 11178, 11738, 10024, 12667, 11262, 11343, 11978, 11983, 11992, 11826, 11061, 10227, 10428, 11077, 11131, 11137, 11280, 11281, 11518, 11551, 11807, 11812, 11947, 12039, 14158, 14729, 14943, 15641, 16273, 16349, 18560, 19269, 19286, 21547, 10343, 14852, 14901, 15352, 15387, 10612, 10638, 10986, 10994, 20181, 11002, 11127, 11128, 11183, 12166, 11342, 11494, 11955, 11673, 11707, 11711, 12190, 18839, 13802, 14595, 12227, 15912, 14007, 13093, 13096, 12981, 13102, 13160, 13194, 13196, 13227, 13242, 13267, 13278, 13281, 13298, 14168, 15161, 23143, 14058, 14766, 13748, 13839, 15603, 14241, 14179, 14239, 14246, 14240, 14255, 14488, 14487, 14245, 14785, 15160, 14703, 14995, 16571, 15172, 15179, 15424, 15713, 15732, 16141, 16284, 16334, 16382, 17387, 17498, 17499, 18498, 16960, 19689, 17815, 19394, 19464, 19650, 19955, 19476, 19477, 23203]
Aha. Maybe this:
Tag.where(name: 'balloon-mapping').length => 4
Tag.where(name: 'balloon-mapping').collect(&:tid) => [10, 3205, 3591, 4032]
Yes, that's the discrepancy:
irb(main):045:0> NodeTag.joins(:node).where(tid: [10, 3205, 3591, 4032]).where('node.status = 1 AND node.type = "note"').count
=> 499
irb(main):046:0> Node.where(status: 1, type: 'note').includes(:tag).references(:term_data).where('term_data.name = ?', 'balloon-mapping').count
=> 499
irb(main):050:0> Tag.where(name: 'balloon-mapping').collect(&:count)
=> [574, 1, 1, 1]
However, notably not for the method
or project
bugs in #7334
irb(main):047:0> Tag.where(name: 'method').collect(&:tid)
=> [14474]
irb(main):048:0> Tag.where(name: 'project').collect(&:tid)
=> [14751]
OK, i fixed the duplicate tags with:
irb(main):053:0> NodeTag.where(tid: [3205, 3591, 4032]).each do |nt|
irb(main):054:1* nt.tid = 10
irb(main):055:1> nt.save
irb(main):056:1> end
however i bet there are more in the DB. We may want to do a bigger project to consolidate these.
Update: yes, there are 28 other instances:
irb(main):063:0> Tag.count
=> 18532
irb(main):065:0> Tag.all.collect(&:name).uniq.count
=> 18504
irb(main):069:0> (Tag.all - Tag.all.group(:name)).collect(&:name)
=> ["fold-up", "gulf-coast", "kite-mapping", "gulf-coast", "brooklyn", "new-york-city", "balloon-mapping", "spectrometer", "farmhack", "white-balance", "fresh-kills", "balloon-mapping", "balloon-mapping", "kite-mapping", "newsletter", "newsletter", "passenger-pigeon", "jamaica-bay", "mobile-spectrometer", "infragram-sandbox", "mailing-lists", "diy-kite", "water-sensing", "carbon-arc", "diy-kite", "diy-kite", "workplace", "emergencies"]
My recommendation now is:
We've narrowed this down a lot, and it has two parts.
question:_____
format tags, which are counted under the questions tab on /tag/____
pages, but if they don’t have their own base tag, are not counted in the topic cards. Example: 8 posts have question:balloon-mapping
but not balloon-mapping
, so they throw our count by 8.My take on (2) is that we should write it down and be aware of it, but not solve it (i can’t think of a clean solution anyways).
And, even after all that digging, I still have a discrepancy of 1 for balloon-mapping
. But I think we are so close that we should essentially be OK with that, as it’s one of the largest tags and that issue is unlikely to affect others. Interested to hear others' take on that too!
Most helpful comment
Hey @ebarry I think this is a more serious issue actually :sweat_smile: I had made changes for this here https://github.com/publiclab/plots2/pull/7609 but I must have messed it up and calculated the wrong count I think :disappointed: Looping in @jywarren Sorry might need some help to fix this up :sweat_smile: