Sunday, March 15, 2009

The Raw Truth: The Actual Temperature readings

What I intend to show in this post is that the raw temperature records are so incredibly bad that there is really no way to correct them. In this post I will introduce you to the state of the raw data.

The way modern science is practiced discoveries are supposed to be repeatable. Joe Blow claims that he has discovered a new phenomenon. Joe publishes his methodology and results in some journal. Sally Skeptical reads the journal and decides to check Joe out, afterall, Joe is a well known blow-hard, at least that is the scuttlebutt at all the conferences. Sally performs the experiment that Joe did and decides that Joe is full of it and then publishes her results claiming that Joe is wrong. Craig Curious is now curious as to who is correct. He repeats the experiment and finds that this one time, Joe is correct. He criticizes Sally's in correct set up. That is how it is supposed to work.

With secular measurements, defined as measurements taken once over a period of time, how can one verify the measurement? With some things, like solar sunspot numbers, one can look at Beryllium-10 (Be-10) which is created by cosmic rays. When the sunspot cycle is active, the sun's magnetic field engulfs the earth and deflects cosmic rays, meaning less Be-10. But, how does one verify the temperature taken in 1926 by a fireman at a Houston, Texas firestation? One can't travel back in time to check to see if he read the thermometer correctly, or even read it at all. So, what is one to do?

One way to double check a thermometer reading from 1926 is to compare it with a nearby town which is in the same environmental setting. Of course one can't do this for a town in a valley compared with a town on top of a mountain, no matter how close they are. So, the place to test the veracity of the temperature record is to compare nearby towns on the Great Plains at similar elevations in similar settings elsewhere. Anyone who lives on the Great Plains knows that, while the temperature might be a few degrees hotter 20 miles away for a day, on average, over the year, it isn't much different 20 miles away than it is here. I need to emphasize that the stations shown below are all from the US Historical Climate Network. What I am showing is the RAW data, not the sanitized and edited data NOAA gives easy access to.

So, let's start with two towns on the flat Texas prairie. Flatonia, Texas and Hallettsville, Texas are two towns that are only 17 miles apart. Here in Texas, it is hot as hell in the summer and mild in the winters. Two locations that close should have very, very similar temperatures. Of course they won't be exactly the same, but the yearly average should be well within a degree. Why do I say this? The average temperature gradient from equator to the pole is .0268 deg F per mile. So, even if these two towns are due north-south, the maximum annual average temperature difference should be no larger than 0.45 deg F. But, with great regularity, ,there is as much as a 4 degree F temperature difference between these two towns.


I must emphasize for the AGW hysteriacs that this is NOT a daily temperature difference. This is an ANNUAL difference. That means that the thermometers are measuring an average of 4 deg F difference throughout the ENTIRE year. This is a tremendous amount of temperature difference.

Here is a plot of the raw temperature difference between these two nearby cities on the Texas prairie.



The data is the raw, unedited data. I prefer this kind of data because there is no bias in it except what is in the raw data. I obtained this data from www.CO2science.org which offers the researcher the actual raw data, untouched by other humans who might or might not have biases.

I want to call everyone's attention to the years 1927-1929, 1934, 1940, 1948-1949, 1955, 1961, 1974-1977, and 1996-1997. In all of these years, the temperature difference between Hallettville and Flatonia were greater than 4 degrees F FOR THE ENTIRE YEAR!!!!

I want to emphasize how huge this annual temperature difference is. It is 8x the January equator to pole gradient for the Northern Hemisphere. The raw data would have us believe that such temperature differences can exist for an entire year, something that I would find ridiculous, but AGW believers might ignore.
So, what do the two stations look like? These pictures are from Anthony Watts web site source
and
source

Hallettsville:


Flatonia"



You can see that both of these stations are rural, yet the output from them does NOT verify the accuracy of the other. The correlation coefficient between the raw data of these two stations is .37. That is worse than mere chance.
Let's look at the two stations subtracted from each other.



This kind of variability in the raw data, which reverses direction doesn't make one confident that one is getting the correct data. The other interesting features is that the data doesn't seem to get better with time. 1996-1997 involve a huge temperature difference, and our instruments are supposed to be better by now.
Lets look at the 1974-1977 period in more detail.



Both stations took a drop in 1973 but Flatonia quickly went up in 1974-1977 but Hallettsville stayed COLD. For the next 3 years there was at least a 4 deg F temperature difference. This can't be real. Such a temperature gradient lasting 4 years would cause huge thunderstorms and winds. No such phenomenon were observed. Because of this, we know that the temperature measuring system is flawed.


Now, some of you are saying. this is rare. This is the exception. I can assure you that everywhere I have looked at the raw temperature data this is the norm. Below are some more stations. I will put them into the format of station A minus station B so that the temperature difference over the short distance can be clearly seen. All of the numbers are ANNUAL temperature differences. I say this to try to get the reader to not think it is a DAILY difference. Daily temperature differences can be big, but they can't be constant over the year for nearby stations effectively at the same elevation.

From Illinois














From Oklahoma




From Mississippi



From New York:



I thought this last one is worthy of looking at the temperatures, given that there is a 12 deg F yearly difference in temperature.



Note that between 1959 and 1971, Walden was tremendously colder than Westpoint just a few miles away. This is, of course, stupid, but it is, of course, the raw data and when the climatologists 'fix' this, they are merely guessing at what the temperature of one or the other town is. Guesses are not data; and one can't possibly fix this data to the level of accuracy required to know how much the earth has warmed or cooled. And, I have to emphasize again, these are yearly average temperatures, not daily temperatures.


A twelve degree difference in temperature means that a cubic meter of air at West Point has 15,600 joules more energy than a cubic meter of air at Walden--for the entire year. That is a huge amount of energy, the energy required to lift 1 kg 159 meters, or 520 feet. Energy always flows from high to low energy places. Such temperature differences, if real, would mean year long winds or year long thunderstorms. None were noted in the meteorological record.

Because of all this, we know that the raw data is crap. It isn't capable of being corrected to know what the quantitative measure temperature change.

14 comments:

  1. You haven't taken into account the effects of microclimate (http://en.wikipedia.org/wiki/Microclimate)on the data. It could, and likely does, account for the temperature differences in the record. Yes, it can be that significant. Here in Oklahoma we often see temperature differences that large and larger between Oklahoma City and El Reno (El Reno is cooler). Why? Because El Reno's weather station sits in a shallow valley.

    Why isn't the variation constant over the years? As you clearly pointed out with both your discussion and your photos, you are comparing two rural weather stations. That means they are operated by local citizens who volunteer for the job. As you can see, the instruments are in their yards. Every time the job is passed to a new volunteer the instruments are moved. Differences in microclimate will be visible in the record.

    ReplyDelete
  2. It is risible to think that this evades the problem I am pointing out. First off, the elevations are almost the same. And the towns are in the same climatic zones, on flat farm lands. I think we can agree that there really isn't 5 deg F difference in real annual temperature over a distance of 20 miles or we should expect to see year long thunderstorms and winds. So, the difference in temperature is some sort of error in the data.

    And if there is, due to microclimate as you say, then it gets even more interesting, queen.
    It really doesn't matter if I haven't taken into account the microclimate issue because there is no way to measure microclimate apart from the actual temperature measurements between two towns. With Hallettsville and Flatonia which temperature are you going to say is affected by microclimate problems? Are you going to change both temperatures to a value you like? How do you decide what value to like? There is no independent measurement of microclimate or urban heat island effect for last Tuesday, so any 'correction' you make to the data is simply a case where you can follow your bias and make the data look like you want it to look.

    The other issue is that the microclimate error, which is in the data, is so large that it overwhelms the tiny signal of global warming. It is not possible to measure a 1 degree F level of warming when the error in the data is 5x plus that level. In statistics, you can't really report your results to a higher precision than the last digit that is changing. Glenn

    ReplyDelete
  3. "It is risible to think that this evades the problem I am pointing out. First off, the elevations are almost the same. And the towns are in the same climatic zones, on flat farm lands. I think we can agree that there really isn't 5 deg F difference in real annual temperature over a distance of 20 miles or we should expect to see year long thunderstorms and winds. So, the difference in temperature is some sort of error in the data."

    If you are so sure that is the case then why don't you mitigate the error by averaging the temperatures of the two sites? In science we like to do our best to minimize measurement errors by taking multiple samples of the quantity in question and averaging them together. Statistics works on the assumption that the truth is somewhere in the middle of the pack.

    "With Hallettsville and Flatonia which temperature are you going to say is affected by microclimate problems? "

    All of them. Every parcel of the atmosphere is a microclimate. And small parcels of air do not behave in the homogeneous manner that makes data analysis easy. It's not at all like viewing the atmosphere on a large scale -- say planetary or synoptic. At those levels features are very smooth and change relatively slowly. Not at all like the small parcel of air surrounding your body. How often do you feel the wind blowing at constant velocity? Do you notice a difference in temperature as you step from the sun into the shade? Even when sitting in one spot do you notice variations in temperature as the air moves around you? The wx stations you are looking at here pick up all those microclimate variations (as well as more stable microclimate features such as those caused by the surrounding environment -- buildings, trees, etc.), yet we're using that data to pick up patterns in large scale features. Knowing how to throw out the noise so that you are left with meaningful information is a science unto itself. Raw meteorlogical data is nearly impossible to read. That's why all the charts and graphs put out by the community show corrected or smoothed data. They aren't "hiding" the truth. They're removing the noise so that we can see the truth.

    ReplyDelete
  4. At some point I will post a note on signal to noise. If the standard deviation is two times larger than the signal you are trying to find, you won't see the signal. When I do a standard deviation on these data sets, it becomes as ludicrous as saying that the temperature has risen by 1.1 deg F +/- 4 deg F. Such statements as that are statistically meaningless.

    So, if you want to claim microclimate, show me how you measure the effect and correct for it other than by simple fiat and faith that you know the truth before you edit the data. Bottom line, if microclimate is causing this much noise in the data set, you can't claim the temperature has risen.

    queen wrote:
    >>That's why all the charts and graphs put out by the community show corrected or smoothed data. They aren't "hiding" the truth. They're removing the noise so that we can see the truth.<<

    That assumes, as I said above, that they think they know the truth before they edit the data stream. You have to know what the noise is before you can remove it. This is where bias fits into the scheme and why Balling and Idso's simple subtraction of the raw from the edited showed that each year brings more editorial input of temperature to the system. Everyone KNOWS that the temperature is rising, therefore editing MUST make that happen. It isn't hiding the truth, it is human bias at its best

    ReplyDelete
  5. OK, Glenn, let's go back to the basics for a moment. In meteorology we look at the atmosphere on a variety of scales. Microscale ranges up to about 100m, local scale will take you up to about 10km, mesoscale up to about 100km. Larger than that is macroscale which is often subdivided into synoptic (hundreds of km) and planetary scales. Microscale phenomena have the shortest life spans, usually on the order of seconds or minutes. Mesoscale phenomena will generally last for many minutes to a few hours. Macroscale phenomena can persist for days or weeks. For a more detailed discussion you can go here: http://weather.unbc.ca/312/notes/notes1.pdf

    Now, let's go back and look at the two stations in Texas you have chosen for analysis. Let's view the photos. These two weather stations are not experiencing similar conditions. The Halletsville station appears to be in an open field with no buildings, pavement, or trees close enough to disrupt the air. True, we can't see what's behind the photographer, but I'm taking the lack of shadows on the ground as evidence that I'm right. Now look at the Flatonia photo. The instruments are parked in the shade of a tree. In fact, there are several buildings and trees in the photo within a few feet of the instruments. A very different environment.

    You tell me that these two stations are 17 miles apart. They could be only 17 meters apart and I would still not be surprised by the differences in the readings.

    Now, let's get back to the meaning of scale. For someone who knows nothing about the composition and nature of the atmosphere it would be reasonable to assume that the temperature gradient between the two stations is smooth. In fact, any other interpretation of the available data would be rejected. And, as you point out, that interpretation would lead to an expectation of constant volatile mesoscale weather conditions. But that isn't the case. Something else must be going on here.

    If the stations were only 17 meters apart I doubt we would be having this discussion. You would most likely recognize the difference in the microclimates as the cause of the discrepancies in the measurements and, of course, no one expects to see mesoscale systems generated by microscale features. So why don't you accept them here?

    And as for the signal to noise ratio -- Do you also disagree with these findings?
    http://map.gsfc.nasa.gov/universe/bb_cosmo_fluct.html

    ReplyDelete
  6. I just had another thought for why Flatonia seems to be consistently warmer than Halletsville. These rural wx stations only report one temperature reading per day. With the widespread use of home computers more and more of these stations are becoming automated, but in the past a human being had to walk outside and make that reading. If the guy in Halletsville was very good about walking outside promptly at 7:00am every morning regardless of the wx conditions while the guy in Flatonia wandered out a bit later (and possibly not even at the same time each day) that is most definitely going to show up in the temperature record.

    Aren't you the least bit interested in why it is each of your difference graphs is strongly skewed in one direction? I doubt very seriously that you are looking at crap data. Something very real and measurable (though not necessarily atmospheric) is going on here. It can be identified and corrected for.

    ReplyDelete
  7. queen of fractal beauty wrote:

    >>OK, Glenn, let's go back to the basics for a moment. In meteorology we look at the atmosphere on a variety of scales. Microscale ranges up to about 100m, local scale will take you up to about 10km, mesoscale up to about 100km. Larger than that is macroscale which is often subdivided into synoptic (hundreds of km) and planetary scales. Microscale phenomena have the shortest life spans, usually on the order of seconds or minutes. Mesoscale phenomena will generally last for many minutes to a few hours. Macroscale phenomena can persist for days or weeks. For a more detailed discussion you can go here: http://weather.unbc.ca/312/notes/notes1.pdf<<<<

    Just so you can know me a bit, you are talking to a physicist by training, a geophysicist by profession. I have done radiative atmospheric transfer physics.

    queen-of-fractal-beauty writes:
    >>>You tell me that these two stations are 17 miles apart. They could be only 17 meters apart and I would still not be surprised by the differences in the readings. <<<

    In that case, as I have consistently stated, the variation becomes noise. I have spent my career figuring ways to reduce noise in seismic measurements. If one doesn't have a model of the noise (that means a way to predict the noise) it becomes impossible to remove deterministically. In seismic data, we can sample the same spot 400 times and that reduces the random noise by the square root of 400. But that does nothing for systematic noise, indeed, that procedure would help it.

    In the case of temperature measurements, you get one sample for each x,y,t. You can't repeat any measurements because you can't go back in time to see if your read the thermometer correctly. Thus, if it reads 10 deg above normal, you can't check to see what the real value was. You either throw the data point out as an outlier or you use it. But if you edit it to some other value, you are merely making the data up to fit your idea of what the temperature was. You can't know you have edited it correctly--no way.

    queen wrote:
    >>>Now, let's get back to the meaning of scale. For someone who knows nothing about the composition and nature of the atmosphere it would be reasonable to assume that the temperature gradient between the two stations is smooth. In fact, any other interpretation of the available data would be rejected. And, as you point out, that interpretation would lead to an expectation of constant volatile mesoscale weather conditions. But that isn't the case. Something else must be going on here. <<<

    Stop acting as if everyone is a sophomore in high school. We all know about microclimates, and we all know that the temperature gradient is not smooth. No one is expecting that. But, what I am expecting is that one should abide by the normal rules of statistical data management and acknowledge that variances in the data, for which you have no deterministic model, becomes noise and interferes with your attempt to know what the climate is doing. Tonight, I posted China data, from the Chinese meteorological bureau. Are you going to try to tell me that 40 deg F is merely a microclimatical difference when that number represents the difference in annual temperature for two towns just tens of miles apart. Do you have any idea what the temperature gradient of that magnitude for an entire year would do??? Where are the thunderstorms? Where is the rain? Where is the wind from such a difference? Is 40 deg F really to be dismissed as a case of microclimate, like Scrooge dismissing Marley as an undigested piece of cheese?


    >>>If the stations were only 17 meters apart I doubt we would be having this discussion. You would most likely recognize the difference in the microclimates as the cause of the discrepancies in the measurements and, of course, no one expects to see mesoscale systems generated by microscale features. So why don't you accept them here? <<<

    What I would accept is that such differences preclude you from measuring climate change.
    queen-of-fractal beauty wrote:

    >>>>And as for the signal to noise ratio -- Do you also disagree with these findings?
    http://map.gsfc.nasa.gov/universe/bb_cosmo_fluct.html<<<<

    I don't see why you brought up this irrelevancy. The WMAP and COBE circled the earth looking at the sky for orbit after orbit after orbit. They could sample the sky hundreds of times and did, in order to get the maps you show. Such a procedure is no different than what we do with seismic acquisition. But, I know this, they didn't get those maps with one pass over the sky.

    You can't claim similarity to their method of acquisition unless you can tell us all here how you go back today and repeat the temperature measurement in Wichita, Kansas, Jan 14, 1958. You have one sample of the temperature on that date. You don't have multiple samples like COBE and WMAP have. So, pray tell, how do you re-sample Jan 14, 1958???



    queen-of-fractal-beauty said...
    I just had another thought for why Flatonia seems to be consistently warmer than Halletsville. These rural wx stations only report one temperature reading per day. With the widespread use of home computers more and more of these stations are becoming automated, but in the past a human being had to walk outside and make that reading. If the guy in Halletsville was very good about walking outside promptly at 7:00am every morning regardless of the wx conditions while the guy in Flatonia wandered out a bit later (and possibly not even at the same time each day) that is most definitely going to show up in the temperature record.

    Aren't you the least bit interested in why it is each of your difference graphs is strongly skewed in one direction? I doubt very seriously that you are looking at crap data. Something very real and measurable (though not necessarily atmospheric) is going on here. It can be identified and corrected for.<<<

    GRM: The problem queen is that the temperatures reverse on occasions. It isn't that one is always hotter than the other because that isn't a correct statement. I work with time series data, so as I said before, you can cease acting like I am some high school kid. Noise comes in all frequency bands. A shift (which is obviously part of the noise system between Hallettsville and Flatonia) is a DC component--zero frequency. In the case of Hallettsville-Flatonia, that component has an amplitude of .53 deg. If you remove that bias, you get a chart that has half the temperature variation above the zero line and half below (or approximately so). The remaining noise is periodic but not predictable. I could do a Fourier transform and learn more about the periodicities of the noise, but unless I have some idea of what happens out there in that rural area, such knowledge doesn't let me remove the noise save by playing mind-reader with God. I can't know for sure if it is Halletsville or Flatonia which is the erroneous station. One can't claim that just because the station is cooler it is better, and neither can you claim that the hotter station is better. Truth is, we don't know.

    Now explain the China data.

    ReplyDelete
  8. I'm not going to ignore your previous comment, but I don't have time to comment on it today. I do, however, have a new thought for you to ponder. Just how "raw" is your raw data supposed to be? I've been so busy looking at the things you want me to see that it wasn't until I was lying in bed last night that it hit me -- these temperatures are way too warm to be an annual average of the actual 7:00am temps reported at these sites. The numbers that you claim are raw have clearly been manipulated in some fashion. Do you know how?

    ReplyDelete
  9. Queen wrote:
    "I do, however, have a new thought for you to ponder. Just how "raw" is your raw data supposed to be? I've been so busy looking at the things you want me to see that it wasn't until I was lying in bed last night that it hit me -- these temperatures are way too warm to be an annual average of the actual 7:00am temps reported at these sites."

    I presume you are speaking of Hallettsville/Flatonia temperatures.
    First off, those are NOT 7am temps. They are annual averages, which would be of the daily max/mins.

    Secondly, you clearly live somewhere up north in Yankee land where it is cold. 70 deg F is not an unusual temperature in a place where one can wear shorts and T-shirts at Christmas time. Do you have any idea of how far south Hallettsville and Flatonia are? They are essentially on the latitude of the Florida panhandle being about 100 miles from Houston, which very rarely gets snow, and when it does, it is merely a dusting.

    You are really really reaching to avoid the obvious conclusion that the raw data is crap--something you yourself effectively admitted when you said that raw data was 'impossible to read'. You are free to continue to believe in AGW, but until you explain this data and the Chinese data and the other data I will post in future posts, please don't act as if you are the only person here being scientific. You aren't. Scientists deal with the data.

    Did you read what the former chairman of the UNC Statistics department said about your criticism? It is today's guest post.

    The data I posted comes from CO2science.org.

    ReplyDelete
  10. "I presume you are speaking of Hallettsville/Flatonia temperatures.
    First off, those are NOT 7am temps. They are annual averages, which would be of the daily max/mins."

    Yes, I do mean those. And historically it was the standard practice that rural reporting stations such as these only report one temperature a day. The thermometer was read at 7:00am. If you have indeed chosen stations that reported max and min temperatures and have done the averaging yourself that's a whole different story. The numbers do make sense for a max/min average.

    "Secondly, you clearly live somewhere up north in Yankee land where it is cold."

    And, no, I don't live "up north in Yankee land" as you erroneously deduced. (Clearly your skills in logic are not what you believe them to be.) I happen to live in Oklahoma City where we too can often wear shorts and t-shirts at Christmas time (though probably not quite as often as you can).

    "Did you read what the former chairman of the UNC Statistics department said about your criticism? It is today's guest post."

    Yes, I did read it. But as I said yesterday I don't have time to seriously comment yet. I'll discuss that with you later.

    ReplyDelete
  11. I grew up in OKC and rarely wore shorts at Christmas time, so we will have to disagree on the proper christmas attire in OKC or I will grant that maybe you are more of a he-man (Or she-woman) than I and can brave the cold better. But just because I am a wimp where it comes to the cold doesn't mean I saw many other of my friends in OKC wearing shorts at Christmas.

    And I also won't buy your assertion that only one temperature per day was taken at a station. Below is a link to an 1891 or so weather reporting form. It clearly takes temperatures several times a day and it records max and min.

    http://www.crh.noaa.gov/ilx/images/east-pia.png

    That form can be found on the page http://www.crh.noaa.gov/ilx/nwshist.php
    That page speaks of the procedure in 1865:



    "February 1, 1865: A Smithsonian weather observing station was established in Springfield, with George M. Brinkerhoff as observer. According to the observations that day, it was 40 degrees at 7 AM, 45 degrees at 2 PM, and 41 degrees at 9 PM; a steady drizzle fell most of the day. Daily weather observations continue through August 1870. These observations were taken on what is now the campus of Springfield College in Illinois."

    They measured the temp 3 times per day and that is what the 1891 temperature record shows.

    I don't know where you are getting your information about the reporting procedures, but I would say you should have done a bit more research before asserting that there was only one reading per day.

    Now, I will grant that some of the old thermometers had maximum minimum bars from which, once a day one could read the max and min temperature. But, such data is not a reading of the temperature at 7am but of the temperature max and min for the previous 24 hours. That is an entirely different thing than what you erroneously describe.

    ReplyDelete
  12. "ust so you can know me a bit, you are talking to a physicist by training, a geophysicist by profession. I have done radiative atmospheric transfer physics."

    And just so you can know me a bit, you are talking to a physicist and meteorologist by training (yes, I went to grad school), a teacher by profession. I may have been out of the field for a few years, but I haven't forgotten everything I learned.

    "In the case of temperature measurements, you get one sample for each x,y,t. "

    Here scale plays in our favor. In fact, it's the basis of your argument. Since we are looking for planetary trends two stations that are only 17 miles apart can be considered to share the components x and y. That means that if measurements at the two stations are taken at the same time (which I think we can safely assume they are) we can treat them as multiple samplings of the same event. And if you are lucky enough to have more than two stations in close proximity you can have even more samples of the point. Instead of screaming that the data is bad because they don't match up you should be averaging them together (I'll even let you choose the weighting function).

    "I grew up in OKC and rarely wore shorts at Christmas time, so we will have to disagree on the proper christmas attire in OKC or I will grant that maybe you are more of a he-man (Or she-woman) than I and can brave the cold better."

    It was colder around here when we were kids.

    "And I also won't buy your assertion that only one temperature per day was taken at a station."

    I didn't assert anything. I asked. It's different. Pay attention.

    ReplyDelete
  13. Then what did I misunderstand? You said that those temperatures were the annual average of the 7am temperatures. They weren't. 7am temperatures would be one per day and then averaged.

    If I misunderstood, I apologize but that was what you wrote. see above

    ReplyDelete
  14. I don't know why I came back to this blog, given how much it annoys me every time I read it because of your fairly blatant biases... but to be honest, your difference graphs bothered me a little bit, so I decided I would figure out the real reason they were different. I also don't trust co2science, so I went to the source:

    http://cdiac.ornl.gov/epubs/ndp/ushcn/newushcn.html

    Then I went to their yearly data, and got somewhat similar patterns to yours. The thing that struck me about these difference patterns you've generated is the stochastic spiky nature: to me, that indicated occasional bad data rather than a physical explanation. If you visually smooth out the Hallettsville/Flatonia graph by ignoring the spikes, it slowly wanders between -0.5 F and 1.5 F, which I would consider explainable by microclimates. One or two year excursions to 5 or 6 F - not so acceptable.

    So, I went to the daily data. And herein is the answer: every one of your spikes that I checked corresponded to a year where there was at least a month or so of missing data. 1973 to 1977 were especially bad examples of missing data.

    Missing data doesn't mess up anomaly measurements too much, assuming that you doing them properly (and that the missing data aren't correlated with the anomaly: if your instrument drops out every time it gets hot, then that is going to mess up the calculation pretty badly!). But large contiguous chunks of missing data messes up calculating yearly average temperatures pretty well.

    Yet another reason to do anomalies and not average temperatures (assuming that the anomalies are calculated from the raw data, and not the annual average temperature data).

    For the Walden Data set, daily data wasn't available, but the monthly data has a column for "flags", and every single month from the end of 1959 to 1972 was flagged "M", which, if I read the documentation correctly indicates a non-standard observation time, which would certainly explain a temperature difference.

    So I think that your major beef with the data in fact comes down to simple data gaps (spikes) and changes in methodology (step functions) and that both kinds of issues are documented in the actual datasets.

    ReplyDelete