Freecodecamp: FCC Weather API randomly returning Japan weather

I think I've identified the issue! It's simply reading from the cache, which contains the Japanese data.

The cache is only used if there are more than sixty requests in its queue (line 50) That means that _sometimes_ it'll return correct results; but _sometimes_ it'll return these, unexpected, results.

How can this be fixed? We have three options...

• Remove the cache
• Make the cache functional, and actually use the latitude and longitude data when deciding whether to use the cache.
  
  
  
  • Reject the request entirely if the server feels overloaded (recommended for now)
  
  

Since this is a help-wanted, I'd love to take a look at this; however, given how this is disrupting projects, and given how I need sleep, if anybody wants to fix this (easiest way is to change the cache to instead reject requests), then please do so!

Okay, I think I fixed the issue. I couldn't test it, though, because I lacked an API key -- and there is, of course, the risk of a DOS given that we never resort to caching.

So, improvements could be made. Should we, perhaps, reject requests where otherwise we'd be sending the Japanese data? This is up to y'all. My fix it temporary and I'll try to work on making it better tomorrow.

I don't think I can make a pull request to the glitch.me API (unless, of course, there's a repository on GitHub which I missed); so someone with write permission will need to merge my changes after testing them with a working API key.

Right, I'm afk at this time; but I realised that the rate limiting is required. I'll fix my code soon so that requests are denied if too many API calls have been made.

Okay, I now think I've created a solution in the short-term. I'm not sure what we want to do about scary data collection though -- the coordinates tend to be quite precise (should we round them for caching purpurses?). We should definitely mark things if they're from the cache, to make it clear it's not 100% accurate.

Anyway, for now, whenever the code tries to access the cache, it'll instead send an error message.

 function getBestCachedData(lon, lat){
+ /*
   var fs = require('fs');
   var obj = JSON.parse(fs.readFileSync('./data/cache.json', 'utf8'));
+ */
-  return obj;
+  return {
+     "error": "This API is overloaded with requests at the moment. Please try again in a few seconds and see if you get a response"
+  }
  }

Hi @joker314 I agree to your solution of rejecting with an error. Can you please make a PR? Thanks a lot for taking the time to investigate this.

@QuincyLarson could you guide who has access to this https://fcc-weather-api.glitch.me/ ? I can be delegated if you are the owner.

@raisedadead I'd love to, but I've looked around and I'm not sure if glitch.me actually supports pull requests -- unless you also have a copy on GitHub or I'm missing something?

EDIT: Of course, the corrected code can be found on my remix.

@joker314 yup you are correct, we will update the project from your remix. Please ignore my previous comment.

I am waiting on Quincy on confirmation of the access to that project. Thanks a lot for your help with this.

@raisedadead - So how long will this temporary fix be in place? What would the ultimate solution for this look like? Is this just a rate-limit issue? If so, it is just going to get worse as more campers create weather projects. Can the rate-limit be increased?

So how long will this temporary fix be in place?

This will not be temporary, with the error it will be known that the issue is due to the rate limits.

This was a wrapper created on top of OpenWeatherMaps API, which was not supporting https I think, and hence conflicted with codepen.

If so, it is just going to get worse as more campers create weather projects. Can the rate-limit be increased?

Yes, you can opt for a paid version by integrating the API directly instead of the glitch wrapper in the challenge.

See https://openweathermap.org/price

@raisedadead It looks like the original code intended for there to be a cache. If this is the case, and if we can smooth out the details as to exactly what we want this to look like to respect privacy, then hopefully we can workaround the rate limiting?

Upon second thought, perhaps it's best to give valid data back instead of an error message. Let's instead change the cache to generate random weather conditions, and set the location to "API busy, try again later for real results"?

What are your thoughts on this, @raisedadead?

As someone who just completed this project and had the issue of receiving Japanese weather data or (most of the time) no data I would like to say that receiving any data is more useful than no data. At least with the Japanese data I could see if my code worked regardless of it showing accurate weather.

Well in theory, we could send random data. But we don t want to get complaints on the data being incorrect.. which I am pretty sure will be a case.

Anyways if it helps lets have the random data.

@QuincyLarson can you please guide with the above request?

I think we should send fake data, but make it clear it's fake. Have the location name be "API Busy, Fake Data", and the weather, coordinates, etc. be randomised in the response. This way, people know why the results aren't accurate; and yet everything will still work for developers.

Thoughts?

If you send fake data, I believe that would defeat the whole purpose of providing the current weather data for a specific location. I like @joker314's idea about figuring out how we want to create the cache and implement it.

Please would you consider pinning recent forum discussions on the project page so that coders know there are issues with the project and don't spend needless time trying to fix something they don't have control over?

BTW, i think the image url of the icon is not returned as described in https://fcc-weather-api.glitch.me/.
It said Images links are included in the JSON under weather[0].icon, which is not.
I noticed that the demo solution wrote css to draw the icon.

Thanks fot letting us know about this potential issue, but the example request amd response do include the desired field. What particular request did you make that didn't have this field?

@joker314 Thx for replying.

https://fcc-weather-api.glitch.me/api/current?lon=39.988&lat=116.3000

{"coord":{"lon":139,"lat":35},"weather":[{"id":803,"main":"Clouds","description":"broken clouds"}],"base":"stations","main":{"temp":28.23,"pressure":1011,"humidity":74,"temp_min":26,"temp_max":31},"visibility":10000,"wind":{"speed":3.6,"deg":230},"clouds":{"all":75},"dt":1499396400,"sys":{"type":1,"id":7616,"message":0.0043,"country":"JP","sunrise":1499369792,"sunset":1499421666},"id":1851632,"name":"Shuzenji","cod":200}

I didn't notice that the example was working fine, so I guess this might due to the location or the special weather?

@Em-Ant It looks like this is an issue with the app running on Glitch. Could you take a look at the cache and see whether this is something we can clear?

I did some quick tests, I believe this is fixed. If anyone is still having problems, feel free to reopen this issue.

@moT01 What kind of tests did you do? The issue is that there is a rate limit per the free FCC account used to make calls to the OpenWeather API. Once those rate limits are reached the Glitch proxy returns the Japan coordinates. The only reason it is not seen as an issue right now, is that this project is now optional in the curriculum, so there are not as many hits to it as there used to be. As soon as you hit the Glitch 60 times in a minute, the following JSON is returned:

{
  "coord": {
    "lon": 139,
    "lat": 35
  },
  "weather": [
    {
      "id": 803,
      "main": "Clouds",
      "description": "broken clouds"
    }
  ],
  "base": "stations",
  "main": {
    "temp": 28.23,
    "pressure": 1011,
    "humidity": 74,
    "temp_min": 26,
    "temp_max": 31
  },
  "visibility": 10000,
  "wind": {
    "speed": 3.6,
    "deg": 230
  },
  "clouds": {
    "all": 75
  },
  "dt": 1499396400,
  "sys": {
    "type": 1,
    "id": 7616,
    "message": 0.0043,
    "country": "JP",
    "sunrise": 1499369792,
    "sunset": 1499421666
  },
  "id": 1851632,
  "name": "Shuzenji",
  "cod": 200
}

Can you reopen this issue, because it is still on my to-do list of projects to fix?

Ahh, yea - I just made a couple quick calls to the api and was able to get correct weather. I suppose I should probably inquire a little more before closing issues.

I started working on a fix right at the start of Hacktoberfest and then got involved heavily with the QA process. The rest is history. I need to revisit this again, because now I have a much better understanding of Node and Express to be able to get a working solution going.

There is a static cache that has only one entry (the location in Japan).

We could fix it removing the rate limiter (I don't know if it's a good idea, since we have only one api key there), or return a rate limit error in case we exceed the requests quota.

Anyway this api project on glitch is owned by @MiloATH , and we cannot edit it without 'forking' it, i.e. creating a new app with a new url.

I have sendt a join request on https://glitch.com/edit/#!/fcc-weather-api using the camperbot account to Milo.

Those sound like good ideas! There's a third option; to fix the cache so that data is actually stored there - or to send random data if the rate limiter is hit.

I fear exceeding the rate limit is not a good idea as the API key could be revoked in such a circumstance, and we might not get results from the API in any case.

@joker314 I was moving in the direction of your third option already.

I've passed on an invite to the glitch project.

The endpoint could be significantly better. I think we should build a separate repo with CI and deploy to something more mature like Heroku (free version). This would allow us to more easily work on the project.

We are not deploying to heroku any more. We will be using Glitch for now. Meaning if there is alternative project we are happy to deploy it under freeCodeCamp's account on Glitch and replace the existing challenge in the Curriculum

@raisedadead @RandellDawson
Hi! Any updates on this?
It would be awesome to see some diagrams with the architecture and the data flow (and eventually the associated filenames)

@Hash2C You can remix the existing Glitch project (shown below) to see and fix the project.

https://glitch.com/edit/#!/fcc-weather-api?path=server.js:1:0

Thanks @RandellDawson, I've been working on it, I hope I can finish a first draft until Thursday

@Hash2C Take your time to get it right. This bug has been here quite some time.

Thanks @RandellDawson, I'll do my best.
I'm going to need to know a little bit more about the current constraints.
I read above that we have a limit of 60 hits per minute, which seems to be the free tier in OpenWeather pricing. Is there a way to increase this limit? Like creating other subscriptions to OW? Are there other free "sources of truth" that we could be using along with OW?

Edit: Also, what kind of delay would be acceptable to deliver? 5min? 15min? 1h? 3h?

Edit2: It seems OW "Weather API data update" is " < 2 hours" as seen in this table
https://openweathermap.org/price
The same table tells us that availability is only 95%

Is there a way to increase this limit? Like creating other subscriptions to OW?

They may also have limits on ip address (not sure), so creating other subscriptions would not help.

Are there other free "sources of truth" that we could be using along with OW?

No sure.

Edit2: It seems OW "Weather API data update" is " < 2 hours" as seen in this table

I am currently developing my own weather project using OpenWeather's free version and I have contemplated just checking if the request is less than 10 minutes from the last request, then show the last returned data for the same lat/lon.

I think we can also update the instructions of the challenge to let the developer know we will send back a special response if a limit is reached, so they can let an app user know the data may not be up-to-date. We would still want to return the same data back we currently are (so as no to break any old projects using the FCC API). We would just be adding an extra property to the response. What do you think?

I created this repo so it get's easier to test locally.
https://github.com/Hash2C/fccWeatherApiDraft

I believe the 3 main use cases (for now) are already covered

• If coordinates are invalid/inexistent , send an error
  else, convert the coords to a convenient format and then we try to hit the cache

• If the requested coordinates are cached
  
  
  
  • If the timestamp is within the acceptable delta: send cached data (1)
  
  
  • else: set mocking data as cached data (in case later OpenWeather api call fails)
  
  

• else: set mocking data as something we decide (in case later OpenWeather api call fails).

• We try to call OpenWeather api.

• If we get the right data, send it, store in cache (2)
• Else, send the mocking data (3)

This flow is hugely open to discussion of course!

There is still a lot of work to do, validations, async stuff, edge cases (tests), etc. but we will gradually get there. I commented the server.js file a lot (don't get scary), I will gradually filter most of that information and ask you for help here so we can discuss each issue.

Just an idea: could we eventually have a data service, that tries to minimize the number of "available requests to the OpenWeather API (or others) that don't get made"? This service would feed, let's say, a MongoDB database - that would be our cache (we could use Memcached but that is probably too much, we don't need that extra speed)

Other ideia: Are there any past usage statistics of this service? If not, maybe we could start creating one, maybe that could be usefull, to try to optimize an eventually possible but very suboptimal solution

One thing I'm surely going to need some help to understand is this Security Issues github is warning me about

(for some reason I totally missed your message!)

They may also have limits on ip address (not sure), so creating other subscriptions would not help.

Good point. Is it worth testing?

Are there other free "sources of truth" that we could be using along with OW?

No sure.

If we are allowed to do this, it may greatly increase the probability of achieving a successful solution.

I am currently developing my own weather project using OpenWeather's free version and I have contemplated just checking if the request is less than 10 minutes from the last request, then show the last returned data for the same lat/lon.

Yes, using cached data right? I brought that < 2h question because I asked previously about the acceptable delay. The longer the delay, the better, so we hit the cache more and we don't have to call the api so often. Started coding assuming we can manage to send data at most 1 h old, just to get it started.

I think we can also update the instructions of the challenge to let the developer know we will send back a special response if a limit is reached, so they can let an app user know the data may not be up-to-date. We would still want to return the same data back we currently are (so as no to break any old projects using the FCC API). We would just be adding an extra property to the response. What do you think?

I totally agree with this idea of giving the relevant info to the developers and letting them choose, it is the path I was considering to be the most adequate too

Are there standard tools for Testing and to make Requests in FCC projects?
For Requests I'm using (just because I decided to give it a try, instead of Axios)
www.npmjs.com/package/request
For testing I don't have much experience but I was thinking about Mocha.
But please let me know which tools better integrate with FCC

One thing I'm surely going to need some help to understand is this Security Issues github is warning me about

The simplest solution is just to run npm audit fix and then commit the updated package.json and package-lock.json. The new packages should not have any breaking changes from the previous, vulnerable ones. However, that assumes the package authors have not accidentally introduced a breaking change, so it's worth manually checking your app after the fixes have been applied.

I have been playing with the OpenWeather api (actually I should have done this from the beginning).
Did we know this? https://openweathermap.org/faq#error401
The relevant part

For FOSS developers: we welcome free and open source software and are willing to help you. If you want to use OWM data in your free software application please register an API key and file a ticket describing your application and API key registered. OWM will review your request lift access limits for your key if used in open source application.

Hello guys, I have been more constrained than expected.
On my free time, I have been studying the OpenWeather api. Unfortunately it is not well documented.
I think I came up with a feasible strategy, using the bbox option, but I'm still testing.
I came up with this idea of creating a document with all the info I came across, the tests I'm doing.

@Hash2C Take your time to get it right. This bug has been here quite some time.

@RandellDawson
You knew what you were saying :heavy_check_mark:

@Hash2C How is your solution coming along?

Closing this issue as the number of users working on this project in general has significantly dropped as it is no longer a required project for a certification. This has lead to fewer instances where the rate limit for the API is reached.