Princeton Election Consortium

A first draft of electoral history. Since 2004

Using predictions in the service of ideals and profit

September 23rd, 2012, 1:00pm by Sam Wang


In everyday life, we use predictions to guide our future actions, from weekend outings (NWS hourly weather) to taking care of our health (EPIC heart study). What do we want from a prediction?

A useful prediction often has the following qualities:

(1) It is precise, allowing us to pinpoint a narrow range of outcomes.

(2) It changes relatively little in the long term, giving us time to plan in advance.

(3) It gives us a true sense of the uncertainty – a sense of knowing what we don’t know. For example, it should capture the true event probability: a 90% prediction should be correct 90% of the time, and a 60% prediction should be right 60% of the time. This distinguishes foregone conclusions from situations that may require action.

When listing these criteria, I am of course thinking of political predictions. But they are to an extent applicable to other types of predictions as well.

At this data-driven site I have given the Obama re-elect probability as close to 90% since the end of July. Here was the prediction then:

and here it is now, after several turns in the race.

History of electoral votes for Obama

What would you do if you had known two months ago that President Obama’s win probability was 90%? One answer would have been to withhold support from your favored candidate, whether it was Obama or Romney. To quote commenter OwlofMinerva:

…with SuperPACs, money can be flexibly allocated. Thus, if it becomes increasingly certain that Obama will be re-elected, we would expect an increasing amount of money redirected into down-ballot races.

In other words, if a win probability is overwhelming, campaign effort is better spent elsewhere. This logic applies whether one is on the losing or winning side.

July readers of the Princeton Election Consortium have had nearly two months to act. Predictions given at this site are based on polls alone. Once an election season starts, two powerful predictors of a final outcome are (a) current polls and (b) knowledge from past campaigns of how much polls are likely to change. The predictions here do not contain any of the econometric assumptions used in political science models or at the website FiveThirtyEight, assumptions that add noise and uncertainty.

Here are two consequences of the knowledge above.

(1) If you want your personal campaign resources – your time, your donations – to affect policy outcomes in 2013, spend them on candidates whose election outcomes are in an intermediate range. For these knife’s-edge candidates, an extra push is most likely to make a difference. A donation to Obama or Romney in July would not have been an effective use of resources.

The ActBlue link shown goes to knife’s edge candidates in the Senate, and to the DCCC. You can see what PEC readers have done. The same logic applies equally to PEC readers who support the Republican side. They have access to strategic thinking in the form of Karl Rove’s Crossroads GPS.

(2) At InTrade, an electronic betting market, in early August a bet on President Obama was trading at $0.58 per $1.00 payoff. This implicitly means a 58% win probability – yet the true win probability, based on polls alone, was 89%. Buying then would have led to a very likely profit now.

As a general rule, InTrade prices express underconfidence relative to true probabilities as dictated by polling medians. This is true even on Election Eve. As I have previously written, they get the direction of the probable outcome correct, but the exact price is  not a quantitative measure of probability.

InTrade prices vs state polls

>>>

And now I give some current probabilities. Do whatever you see fit, whether it be to go to Crossroads GPS for Republicans or ActBlue for Democrats. As for other possibilities…we all know that gambling is wrong.

President Obama re-elect probability: 89%. InTrade probability: 70%.

Senate to remain Democratic-controlled: 88%. InTrade: 58% 82% (100% minus Republican-controlled share price)*.

House to remain Republican-controlled: 26%. InTrade: 80%.

If you have other qualities that are useful to have in a prediction, give them in comments.

Update, 5:00PM: In comments, Terry makes a valid point that the House prediction feels “softer” than the other ones. Indeed this is true – the probability itself has a larger uncertainty. The generic Congressional ballot is crazy-making from a predictive standpoint because unlike the Presidential and Senate races, it is an indirect measurement. I have examined its track record and uncertainties, most recently here.

The situation recalls to me a remark of Amitabh Lath weeks ago, in which he made a comparison to high-energy physics. Evidence leading to a particle’s discovery such as the top quark or the Higgs particle often come in multiple channels of data, some of high sensitivity and others of low sensitivity. The House prediction is lower-sensitivity than the other two predictions (though not nearly  low as “fundamentals”-based predictive models).

*Update, September 24: In comments, Roestigraben points out that the InTrade “Democratic” contract does not count Independents expected to caucus with Democrats (i.e. Sanders, King). I have replaced the probability with a correct figure calculated as 100% minus the Republican-control contract probability.

Tags: 2012 Election · House · President · Senate

47 Comments so far ↓

  • Amitabh Lath

    I absolutely agree with your major point. I understand why others add economic data, etc. But sticking to the polls is the most defensible approach.

    Minor quibble: in point 2 you say that the prediction changes little over the long term. This can happen if opinons are not changing, but also if polling is sparse.

    You could plot the x-axis not in real time, but in units of equal number of polls (poll-time).

    But anyway, it’s a moot point since now everyone and their uncle seems to be polling away madly.

  • Iseeurfuture

    Sam,
    Your simply the BEST. Thanks.

  • Tapen Sinha

    Doyle McManus wrote yesterday in LA Times:

    “Only six weeks to go in the presidential campaign, and the public opinion surveys have developed a case of the jitters. Last week, one respected poll reported that President Obama had opened an eight-point lead over Mitt Romney, but another reported that the race was dead even. Other surveys were scattered in between. What’s a poor voter supposed to believe?

    I consulted three smart pollsters — one Democrat, one Republican, one nonpartisan — and they all offered the same advice: Calm down. It’s not as crazy as it looks. Yes, Obama has taken a lead, but only a modest lead, not one big enough to prevent Romney from closing the gap if he can only find the right ingredients.”

    http://www.latimes.com/news/opinion/commentary/la-oe-mcmanus-poll-election-swing-voters-20120923,0,6224480.column

    On a personal note, in August 2008, I took a bet with another person that Obama will win. It was a USD1,000 bet. I have taken another bet with her in August 2012 of the same amount that Obama is going to win. She has fallen for the classic “I need to get even” trap that most naive investors do.

    Tapen

    • wheelers cat

      Tapen, I will disagree on the basis that the “right ingredients” no longer exist.
      Dr. Wang has Obama’s win probability at 88% only because he is allowing for the possibility of a black swan event that swings the race to Romney. What could that possibly be? I cant think of one.
      Nate Silver has Obama’s win probability at 95.8% in the (polls only) nowcast. Nate does not allow for the possibility of a black swan, and so …..in theory…..Romney’s win probability could approach zero.
      mirable dictu.

    • bks

      wheelers cat: Black swans are not so hard to come by. For example suppose Obama has a stroke before or during the debates?

      –bks

  • Ebenezer Scrooge

    Other reasons to have a prediction? Say you’re planning a celebratory election party. You don’t want to be surprised.

  • Olav Grinde

    A question on a minor point… I understand you currently rate Obama’s chances of re-election at > 90 %. However, looking at the histogram of all possible outcomes, I don’t see any bar giving Obama less than 270. I don’t see any visuals indicating any chance of a Romney win at all.

    http://election.princeton.edu/todays-electoral-vote-histogram/

    • Steve_OH

      @Olav,

      The histograms show the results of an election held _today_, while the re-election probability is for an election held in November. That’s the difference.

  • Olav Grinde

    Ok, here is a very cynical double question:

    Assume accurate medians in state-wide polls, how many votes would the GOP have to “steal”, and in which swing states, to most easily clear a path to 270 Electoral College Votes?

    (For the sake of argument, let’s limit ourselves to swing states that have Republican governors and secretary-of-states.)

    And how do those numbers compare to the actual purges of voter rolls that have taken place in these respective GOP-controlled states?

  • Olav Grinde

    …so, in essence, if you can’t win by other means, you need reliable predictions in order to know how much you have to cheat to close the gap.

    • David Kline

      This site does provide information about which states would provide the most value per stolen vote, if that were the question one was asking.

  • David Kline

    Sam, to answer your questions about what other attributes are necessary for a prediction to be useful, I’ll point to one thing that’s critical. It’s implied by your criteria above, but could be stated more explicitly. The prediction (or, in a broader sense, “information,”) has to have the potential to change a decision that you care about. I point this out because there is definitely the possibility of great, accurate information being worth very little because it never had the potential to change your decision. Looked at this way, one of the aspects that shows the value of your information is what you point out above–that it provides a guide to where money can be most usefully deployed, and has for some time. On the other hand, if we were to ask how much value it would be to improve on your data from this point, e.g. by getting more state polls or doing more research to sharpen your methods for weighting them, you’d find that more information beyond what you have now would have close to no value.

    • Felix

      The other value of prediction is sanity! This is what Sam gives us (at least Dems this time around) on this site. If the other outcome seems disastrous beyond believe, then I want EVERY prediction, every poll, every piece of information that even a ‘black swan’ will no long be able to change the outcome.
      I bet this is what keeps many of us coming back here!

    • Sam Wang

      Ah, but will you come back when the news goes the other way? It did in 2010, and it will again someday.

      Anyway, the wind may be at your back, but this does not excuse you from working the sails.

    • Felix

      its that trust thingy.
      Most people who will read the comments here do understand the difference between a 99% prediction and the actual outcome!

    • wheelers cat

      “Anyway, the wind may be at your back, but this does not excuse you from working the sails.”

      maybeso, but I bet we can douse the spinnaker and run for quite a bit. Evolutionary electoral demographics is going to make for forty years in the wilderness for the all white GOP.

  • Terry

    Well, these are rather pleasing probabilities particularly to the extent that they control for poller error e.g. recent Gallup results (do they?).

    I have learned a lot on this site in terms of the statistical model and its assumptions; and its rather mature discussions as how to handle ‘noise’ and, specifically, the preference for live contacts including cellphone constituents.

    Polling has too-long been an art and not a science; if not an outright manipulation.

    I have long been a fan of Nate Silver. However, I now see him more as a politics ‘color man’ (in the football sense). His opinions are way above the typical pundit. He offers intelligent commentary – not always supported by his ‘nowcast model’.

    I find his ‘Nov 6′ model to be contrived due to his artifical convention ‘bounce’ constraint which amounts to more noise/error variance.

    I think he will have some explaining to do after the election as to the predictive utility the latter model.

    For the social sciences, it is a bit disappointing not to be able to add more viariables to explain the most variance. The lesson that I lam learning (I think) is that most of these constructs are already accounted for in a reliable and valid polling strategy.

    Ultimately the numbers will not fail and Obama will have won and Dems will have retained the Senate. My gut doesn’t yet agree with the House projections….maybe when we have some more polling data.

    • Sam Wang

      I completely understand about the gut feeling re the House. The generic Congressional ballot is far more crazy-making from a predictive standpoint.

      This all recalls to me a remark of Amitabh Lath on this site, in which he described the particle physics community’s multiple streams of evidence leading to the Higgs particle: a low-noise channel and a high-noise channel, which constitutes softer evidence. I might put the generic ballot as being toward the latter category compared with Senate/Presidential polls.

  • Tapen Sinha

    @wheelers cat
    mirabile visu – note the juxtaposition of that column and my betting. If I believed like McManus does, would I be betting? Unless my beliefs give me strong odds in my favor, I do not bet. In recent years I have made about $10 per $1 bet in the Oscars sweepstakes. I bet about $100 per year.

    Somebody asked me in an email what I do with the money. I donate them all to the Orphans of AIDS charity whatever I win.

    Tapen

    • wheelers cat

      Im disagreeing with McManus, not your mad betting skills.
      sorry if that wasnt clear.

  • Anbruch

    There is one element about dollar allocation that I would dispute: Money given to Romney or Obama is also used in GOTV efforts that benefit downticket races. Funds may not be as efficiently allocated as giving the money directly to the campaigns of downticket races, but there is more likely to be carryover from establishing the Obama campaign field offices to Senate and House races than from money given to Senate and House candidates to each other or to Obama. The same applies, mutatis mutandis, to the campaigns on the GOP side. This is not to dispute your point, and I have shifted most of my donations into the competitive Senate and House races but I have not zeroed out giving at the presidential level yet either.

    • Ebenezer Scrooge

      This is a reason for giving more to the close Senate races in bright blue or red states–there won’t be a Presidential GOTV there. Hence, I’m not giving as much to Kaine or Nelson. Obama’s helping them plenty. Heitkamp or Warren are a whole ‘nuther matter.

  • Olav Grinde

    @Steve, David: Thanks!

    Another quality is that a prediction come from someone we trust. I know that is key for me — as there is no shortage of them.

    A second, for me as a layman, is the clarity of reasoning behind it. I think there is a distinction to be made between this and the actual methodology.

    Perhaps a third, if we’re talking about predictions influence society at large and policy makers, is penetration. And by that I mean that the predictions, the reasoning behind them, the recommended action, and the consequences indicated, actually reaches the audience in question — in a form that audience is able to absorb.

    A fourth, perhaps, is that this prediction is able to gain prominence over competing ones, especially those put forward by experts who are on the payroll of someone with a specific agenda.

    • wheelers cat

      yes Olav, penetration and competition!
      Nate Silver is the preeminent forecaster of the known blogverse….so he has great penetration and influence.
      But observe!!!
      The next time the forecast goes over 80% Nate will be attacked again on twitter and in blogspace by conservative pundits. I suggested he could perform an experiment by having the nowcast come up as the default.
      I think it is bidding theory– 2:1 or 3:1 odds arent so bad….but 4:1 or 5:1 become non-cost-viable.

    • Sam Wang

      One difficulty here is how to define trust. I attempt to generate trust with relative openness about the methods. But it is also clear that I have a point of view. Does that change the level of trust?

      In regard to audience reaction, I am cautious towards that idea. To take hurricane prediction for an example, the proof is in the verification of the predictions, as opposed to consumer reactions. Also, a track record takes a long time to accumulate in an area like political commentary. It might be that benchmarks would be a useful means of achieving this criterion. Pollyvote, cited elsewhere, is one such source of benchmarks.

    • wheelers cat

      Dr. Wang, that gets into Social Network Theory and the concept of trusted sources, influence and opinion weights, doesn’t it?
      A way to measure that is connectivity and page clicks.

  • Amitabh Lath

    Thanks for the shoutout to particle physics. I agree that the generic congressional polling has information, but as you said, it is difficult to tell how sensitive it is.

    Naively, I would think that a generic congressional poll would have very little predictive power, simply because most of your data would be from “safe” districts. There are several hundred safe seats, and only a few that are truly up for grabs. Even a large statistics poll might not catch anyone in a contested district.

    But then you showed that generic house polls correlate with outcomes. So obviously the generic poll is telling us something. Maybe this channel has good analyzing power after all.

  • Tapen Sinha

    Sam notes: “One difficulty here is how to define trust. I attempt to generate trust with relative openness about the methods. But it is also clear that I have a point of view. Does that change the level of trust?”

    Yes it does. I bet you my last peso that you will not find regular listeners of Rush on this website by the droves.

    I suspect most readers of this website are looking for confirmation of their own beliefs about how well Obama is doing.

    In an ideal world, if polling were as scientific as particle physics, then it would not matter who is saying it. An electron is an electron is an electron. Or so we thought as the beginning of the last century. Hell, even now we do not know much about dark matter, Higgs boson (may be), and a whole host of things in physics!

    Tapen

    • Sam Wang

      All the more pressure to examine one’s own assumptions carefully, and to be as transparent as possible.

      We post the MATLAB code and data sources. Anybody who wanted to could reproduce the work described here, and/or check what we are doing. And from my correspondence, I assure you that we do get checked.

    • Amitabh Lath

      Hi Tapen, actually physics (the experimental side) does have a lot in common with political polling. As the initial data comes in and gets analyzed, you have no idea what’s going on. Speculation runs rampant. Then several layers of calibrations, corrections, and other refinement gets applied, and eventually something starts to peek out. Or not.

      I’ve seen this happen with the Higgs boson (the 2011 data was really murky, then the 2012 data came in and polish, polish, polish and eventually… 5 sigma). Same thing, 17 years ago, with the top quark.

      I have also seen it happen over and over again with false positives. 2 sigma fluctuations that eventually die away as more data pours in.

      I suspect if you could ask JJ Thomson about the electron, he would recount a similar story about his “cathode ray” discovery.

      Physicists tend not to go after problems which do not eventually yield clean signatures. If physicists were in charge, they might just shut down polling because of the irreducible systematic uncertainties.

      This is why a biologist like Sam is needed to run this site. In spite of the murky data and complicated systems, he has identified the one quantitative statement one can make.

    • Sam Wang

      Also…to anyone who thinks that discovery in the physical sciences is cut-and-dried, I strongly recommend the book How Experiments End, by Peter Galison. Even with careful and rigorous methods, discoveries take a while to be polished and agreed upon. Galison has a brilliant chapter describing years of work in which experimental physicists attempted to measure the gyromagnetic ratio of the electron. Without the idea of particles having a quality called spin, the theoretical value was off by a factor of two. Experimental measurements were clustered near the pre-spin value. With time, experimental measurements started drifting toward the new value. Amusingly, one person who was led astray was Albert Einstein, in a rare foray into experimental work.

      A challenge to an analyst of quantitative data is to care enough about the subject to maintain high intellectual standards, yet not be blinded by preconceptions.

  • Joel

    I think the fundamentals models are only useful for the longest-term forecasts. This is kind of akin to the “pythagorean” model in baseball. At a certain point, the RS-RA for the year begins to be less predictive than actual W-L record. And that’s not a very high bar to clear. Polls are even better.

  • Amitabh Lath

    It is interesting to see the history of measurement of a physical quantity. For example, the best determination of the speed of light, vs year of measurement.

    Rather than a smooth set of points approaching the current value, what you saw was a bunch of measurements clustering around some value, and then a jump to a new value.

    There is a lot of pressure to agree with previous measurements (aka “conventional wisdom”) even when you have much better techniques.

  • Tapen Sinha

    @Amitabh and @Sam

    At least a sub-atomic particle behaves the same way whether I am looking at it or not (maybe!). Forecasting voters is trickier as additional information can shift voter preference.

    I do greatly appreciate Sam’s open approach (as opposed to Silver who talks about THE model in third person – as if the model itself is some other being – and never clearly stating the actual model!).

    Tapen

    PS: I was thinking about how the oil drop experiment data was cooked by Millikan. I am still a century behind in physics.

  • Matt McIrvin

    So, when does your prediction uncertainty start tightening up on the basis of not enough time left for a large fluctuation? Oct. 1?

  • pechmerle

    “Ah, but will you come back when the news goes the other way? It did in 2010, and it will again someday.

    Anyway, the wind may be at your back, but this does not excuse you from working the sails.”

    Put other ways, “this is why they play the game,” despite the Vegas line. (Cal vs. USC – Cal beat the Vegas line by 1/2 point yesterday while losing by 16 — one could have made a little money on that). “This is why they run the race,” despite the odds of 20-1 on the tote board for your horse.

    More seriously, if SW’s type of analysis shows the result is going to be unpalatable, the analysis is still a guide to action. It tells you what other races to focus your time or money on; it tells you whether something other than electoral activity may be called for. And down at the trivial level, it tells you to bet against your preferred candidate — you can always hope you lose the bet! (See also the terrific Hemingway short story, “Fifty Grand.”)

  • Roestigraben

    One thing that needs to be kept in mind when comparing Intrade’s Senate control odds to your model’s numbers is that the rules for this specific contract don’t cover a common-sense interpretation of what “control” means. The SENATE.DEM contract that’s now trading at 58 will only pay off if Democrats retain a “pure” majority without independents (Sanders and likely King) – that is, 50 seats under an Obama presidency and 51 seats under Romney. So what people are betting on there is what most people would consider at least 52-seat Democratic majority.

  • Peter D

    Sam,

    Election day is now 43 days away. Currently the MM is 4.36, or 1.98SD (MMSD = 2.2). tcdf(4.36/2.2,3) = 0.9291.

    However, you’ve indicated before that poll error begins to converge 40 days out. Do you have any guidance on what this implies for the MMSD going forward? Is there a mathematical description for this decay (http://election.princeton.edu/wp-content/uploads/2012/08/days-forward-calculation_500px.jpg)?

    Peter

  • Kevin Kesseler

    Professor Wang,

    As a professional modeler myself, the things I look for in my models above all else are testable hypotheses. My current efforts have been in constructing rule based models of protein-protein interaction systems with the goal of producing simulated experimental results. Obviously this type of problem is more amenable to producing predictions which can be experimentally verified than your model here (or Nate Silver’s model at 538), but I think that testability is an important criterion for predictions from any sort of model.

    Unfortunately “weather forecast”-type models can only be empirically verified using large data sets (and we can’t really set up a time loop between now and the election and run through it a couple of thousand times to generate sufficient empirical data), so the most obvious predictions you make can not really be tested, but it seems like some aspects of your model might be possible to test.

    In particular, both you and Mr. Silver suggest that the numbers in the betting markets do not accurately reflect the odds—in which case there would be an arbitrage opportunity. Now I don’t know enough about hedge fund management and commodities trading to determine how to test the accuracy of an election model using those markets (or if the structure of the markets would even allow the necessary trading), but I would think that it is possible to come up with at least a hypothetical metric to test any type of predictive model.

    For instance, supposing one had access to the daily volume and price data from one of the electronic markets the results of the following scenario could be calculated:

    Imagine being the sole broker of all of the trading on the market in question and follow some hedging strategy depending on your daily probability estimate and the price in the market. To give an example, suppose that each day you took the total value of the bets placed, divided it according to your predicted probability of an electoral victory rather than the current price and used it to purchase shares in the market on that day.

    On election eve, the value of all of the shares—at the final pre-election prices—would give, in my opinion, a good metric of the model. Would this show a profit in comparison to the money collected, including interest (to weight earlier predictions more according to the time value of money)? Such a measurement would provide at least a hypothetical test of the prediction that there was an arbitrage opportunity in the market.

    I’m sure that you or your readers could come up with improvements to my naive attempt at a metric, but I hope you get the idea. I would be very interested to see a comparison of your meta-analysis and Mr. Silver’s model in this sort of metric.

    I have read some of your criticisms of 538′s model (I have been following 538 for some time) and while your points are well taken, I believe that most of what you point out is merely the result of different (but still valid) methodological choices resulting from differing goals and personal preferences. If I were, say, a Koch brother trying to decide what to do with my money, I would base my decision primarily on your meta-analysis (while considering what Mr. Silver’s results meant as well), but if I wanted to take advantage of arbitrage opportunities on Intrade, I’d bet ;-) on 538′s numbers (while looking to see if I would have done better or worse with yours). As someone who is a quant geek that is interested in politics, I’ll keep giving considerable weight to Nate’s analysis, but I’ll give serious consideration to what you have to say as well.

    Disclaimer: There are two factors, one personal and one professional, which bias me towards Mr. Silver—I grew up in the same hometown as Nate (East Lansing, MI) and in my recent work I’ve been employing a methodology which has more in common with the model at 538 than it does with your meta-analysis.

  • Till

    I disagree somewhat with your SuperPAC argument: the assertion that someone wanting to make the most difference would use his money to support a candidate in a close race is correct, of course. However, the motivation of donors (especially large donors) is not only to change the outcome of a race, but also to give to the winning campaign – which is why many “institutional investors” like insurance or defense companies often give to both sides. Hence, if the probabilities start to strongly favour one candidate over the other, donations may dry up for the presumptive loser, but not necessarily for the presumptive winner (and such a shift would be significant as well). Not all SuperPAC donors remain anonymous, most likely because they expect a return for their support.

    • Kevin Kesseler

      For “industry” or “institutional” superPACs, you might be right, but for investors like the Koch brothers and groups like Crossroads GPS, they are ideologically committed to supporting one side. I believe that you will start to see this money start flowing downticket rapidly.

    • Sam Wang

      That was my very point!

    • Kevin Kesseler

      Professor Wang,

      I was disagreeing with Till and agreeing with you. Personally, I think that the superPAC money (which downticket opponents don’t have a prayer of matching) is much more dangerous at the state or local level and I expect that the deep pocket donors know this as well.

    • Till

      Just to clarify, I did not mean to say that a race becoming less competitive does not lead to (rational) decisions of allocate resources elsewhere, like downballot races. I only meant to say that such a shift may not be symmetric; because there are also incentives for an important group of donors to support a candidate who is clearly winning, especially if they were previously supporting the other candidate. It all depends on whether someone just wants a candidate to win, or whether someone wants something in return from the candidate. There may in fact also be an incentive to support a candidate whose chances are very slim, because the number of other supporters competing for favours from the candidate will be reduced (an example being Foster Friess donating to Rick Santorum during the primaries: Santorum’s chances were always small, but this meant that Friess was the single most important supporter). All of these effects may be small compared to the “anonymous rational” supporter seeking to have the greatest impact on the race itself, but they may suggest that just “following the money” to infer perceived probabilities is not as simple.

  • Kevin Kesseler

    Till,

    There is also a strategic reason to stick with Romney (or any presidential candidate who is very likely to lose)—to prevent an “enthusiasm collapse” that could hurt downticket races.

Leave a Comment