a
a
Weather:
No weather information available
HomeMeteoPagination Is Lying to You: Historical Weather Date Ranges Explained

Pagination Is Lying to You: Historical Weather Date Ranges Explained

Pagination Is Lying to You: Historical Weather Date Ranges Explained
Pagination Is Lying to You: Historical Weather Date Ranges Explained

The historical weather endpoint goes back to July 2008. That’s nearly 17 years of data, and for most queries you’ll never hit a wall — until you do, and you won’t know it happened.

The problem isn’t the data. It’s that the API’s date range behaviour is easy to misread in ways that return a successful 200 response with subtly wrong results. No error. No warning. Just quietly truncated or shifted data that breaks your analysis downstream.

Here’s exactly what’s going on and how to defend against it.

How the Date Range Parameters Actually Work

The historical API takes two parameters: date (start) and enddate (end), both in YYYY-MM-DD format. What looks obvious turns out to have a few edge cases worth knowing:

  • The boundary is inclusive on both ends. A query for date=2023-01-01&enddate=2023-01-31 returns data for all 31 days. Sounds obvious, but if you’re slicing month boundaries programmatically and your loop adds one day to the previous end date, you’ll double-count.
  • Requests spanning more than roughly 35 days get silently truncated to that window in a single call. Not an error — just a cap. The response closes at the cutoff and you have no way to know the remainder was dropped unless you check the last date returned against what you requested.
  • Querying before July 2008 doesn’t throw an error. The API clips the range to the available archive floor. Request June 2008 and you’ll likely get a partial response starting in July 2008. This one trips up backfills the most.

The fix for the truncation problem is straightforward: always read the last date value in the returned weather array and compare it to your requested enddate. If they don’t match, paginate.

A Minimal Pagination Loop That Actually Works

Here’s the pattern I’d reach for in Python. No library magic — just explicit date arithmetic and an honest check on what came back:

import requests
from datetime import date, timedelta

API_KEY = "your_key_here"
BASE_URL = "https://api.worldweatheronline.com/premium/v1/past-weather.ashx"

def fetch_historical(location, start: date, end: date, tp: int = 24):
    """Fetch historical weather for a date range, paginating in 35-day chunks."""
    all_days = []
    cursor = start

    while cursor <= end:
        chunk_end = min(cursor + timedelta(days=34), end)

        params = {
            "key": API_KEY,
            "q": location,
            "date": cursor.isoformat(),
            "enddate": chunk_end.isoformat(),
            "tp": tp,
            "format": "json",
        }

        resp = requests.get(BASE_URL, params=params, timeout=15)
        resp.raise_for_status()
        data = resp.json().get("data", {})

        if "error" in data:
            raise ValueError(data["error"][0]["msg"])

        days = data.get("weather", [])
        if not days:
            break  # shouldn't happen mid-range, but don't loop forever

        all_days.extend(days)

        # Advance cursor past the last date actually returned, not past chunk_end
        last_returned = date.fromisoformat(days[-1]["date"])
        cursor = last_returned + timedelta(days=1)

    return all_days

The critical line is cursor = last_returned + timedelta(days=1). Using chunk_end + timedelta(days=1) instead would silently skip days if the API returned fewer days than requested. Advance past what actually came back, not past what you asked for.

The Interval Parameter Changes What You Get — Not Just How Often

The tp (time period) parameter accepts 1, 3, 6, 12, or 24. This isn’t just a data density toggle. At tp=1 you get hourly observations; at tp=24 you get one aggregated daily record. Same date range, completely different shape.

What catches people: tp=1 over a 35-day window returns roughly 840 records. Fine in JSON terms, but if you’re writing to SQLite or a dataframe your column expectations need to match the interval. Hourly records include fields that simply don’t appear in the daily record — chanceofrain as an hourly series is derived differently from the daily chanceofrain, which is the max across the day’s hourly window.

For ML feature pipelines, this matters more than it looks. Training on tp=24 and accidentally fetching tp=1 for inference doesn’t just give you more rows — it gives you a different statistical distribution for the same field names.

What Happens at the Archive Floor (July 2008)

Querying before the archive floor is the quietest failure mode. The API won’t reject the request. What you get back depends on how far before July 2008 your start date falls.

Concretely: a request for date=2007-01-01&enddate=2008-12-31 will return data, but only from the available archive start. The returned weather array begins at the floor, not at your requested start. If you’re expecting 731 days you’ll get fewer, and any index assumptions downstream break quietly.

Defensive check:

ARCHIVE_FLOOR = date(2008, 7, 1)

if start < ARCHIVE_FLOOR:
    print(f"Warning: clamping start from {start} to {ARCHIVE_FLOOR}")
    start = ARCHIVE_FLOOR

Clamp before the request. Don’t rely on the API to handle it gracefully for you.

Marine Historical Has a Different Floor

Worth calling out explicitly because it’s a separate dataset: historical marine weather is available from January 2015, not July 2008. That’s a meaningful gap if you’re building something that switches between standard historical and marine historical based on query type.

The marine dataset ingests different source data — wave model outputs, buoy observations, tide gauge records — rather than being derived from the atmospheric archive. Same API key, same general request shape, different endpoint and different floor. Mixing up the floors in a backfill loop produces the same silent truncation problem described above, and it’s harder to spot because you’re not expecting marine data to run as deep.

Rate Limits and Backfill Strategy

On the free tier you have 500 calls per day. With 35-day chunks, a full backfill from July 2008 to today is roughly 180 chunks per location. Spread across two days that’s manageable. It becomes a problem if you’re backfilling multiple cities in parallel and burning through your daily allowance before you notice.

The practical approach: backfill one location at a time, checkpoint to disk after each successful chunk (write the raw JSON, don’t just aggregate in memory), and restart from the last saved date if the script dies. Saving raw JSON also means if the field schema changes later, you can re-parse without re-fetching.

Even on a paid plan with higher limits, the 35-day chunk strategy still makes sense. Smaller chunks mean a smaller retry surface when something goes wrong at chunk 47 of 180.

Validating Your Data After Backfill

Three checks worth running before you treat a backfill as complete:

  • Date continuity. Sort all returned date fields and diff consecutive rows. Any gap larger than one day means a chunk was dropped or a pagination step was skipped.
  • Field completeness. Pick a known-present field (maxtempC for daily, tempC for hourly) and count nulls. Sparse observation coverage in the 2008–2010 period means some fields genuinely have no data near the archive floor — that’s real, not a bug. A cluster of nulls mid-range is usually a failed chunk that got partially written.
  • Reasonable value ranges. maxtempC outside −80 to 60 for land stations, or windspeedKmph above 400, almost always means a unit mix-up or a parsing error rather than real data.

Any API returning weather back to 2008 is stitching together model reanalysis data with observation records, and the seams show up as coverage gaps and field sparsity at the edges. Treat the validation step as mandatory, not a nice-to-have.

One decision worth making before you start a large backfill rather than halfway through: tp=1 or tp=24. Changing your mind means re-fetching everything. The storage cost of hourly data across a decade and a dozen cities is real, but it’s smaller than the analytical cost of building on daily granularity and then discovering you needed hourly.

The post Pagination Is Lying to You: Historical Weather Date Ranges Explained appeared first on Weather Blog.

No comments

Sorry, the comment form is closed at this time.

Translate »