Cancellation of the Archon Genomics XPRIZE: A Public Debate

Cancellation of the Archon Genomics XPRIZE: A Public Debate

Dr. Peter Diamandis, Chairman & CEO, XPRIZE

Opening Statement

XPRIZE is the global leader in the creation of incentivized prize competitions. We are a non-profit organization, whose mission is to bring about radical breakthroughs for the benefits of humanity, thereby inspiring the formation of new industries and the revitalization of markets.

An XPRIZE is a large-scale, monetary award given to the first team to achieve a specific goal, set by XPRIZE, which has the potential to create or catalyze an industry and thus positively impact humanity. So rather than awarding money to honor past achievements or directly funding research, an XPRIZE incites innovation by tapping into our competitive and entrepreneurial spirit.

An XPRIZE must set an audacious but achievable goal that captures the imagination of the public, spurs innovation, and accelerates the rate of positive change across the globe.

At XPRIZE, we take big risks. We launch an XPRIZE with the understanding that teams may either fail to achieve the objectives set out in our prizes or that the prize objective may be come irrelevant due to the economic incentives of a marketplace. The Archon Genomics XPRIZE was the first XPRIZE to be cancelled based on market incentives outpacing those of the competition. If we only launched prizes we knew would succeed, we wouldn’t be taking enough risk to reach the audacious goals we set for ourselves and humanity.

1. What should be the procedures and circumstances for cancellation of an XPRIZE?

An XPRIZE is successful only when it creates or catalyzes an industry by addressing key market failures that prevent innovation in a given field. However, when or if those market failures are being solved by current industry developments, an XPRIZE is no longer required to create breakthrough innovation. An XPRIZE cannot be awarded to honor past achievements or celebrate an accomplishment that could otherwise occur without the prize. We are committed to creating and catalyzing new markets and only where traditional market forces are not working properly. When market realities make an XPRIZE irrelevant to its intended goals (and there is no binding agreements with any teams), we must as a non-profit organization return our donor’s funds.

When launched in 2006, the Archon Genomics XPRIZE, sought to address the many market failures in the whole genome sequencing industry of the day, which include poor accuracy, high cost, low quality, and long processing times of genomic sequencing technologies. However, over the seven-year life of the Archon Genomics XPRIZE, these technology performance issues were being solved at a rate that outpaced all technological and economic expectations. By 2013, the Archon Genomics XPRIZE was no longer relevant for driving innovation in the market. Also, our standard process of moving a competition forward is to register teams, then sign a binding Competitor Agreement in which the XPRIZE Foundation and the Team(s) agree on the final rules. By 2013, the prize had neither officially registered teams nor any Competitor Agreement signed with teams. Therefore, the XPRIZE Foundation made the decision to cancel the prize.

This was the first time XPRIZE has ever cancelled a prize and there was no precedent established in terms of processes or procedures. The XPRIZE therefore, followed a protocol it established for this prize. XPRIZE consulted with:

  • Members of the prize’s Scientific Advisory Board, Co-Chaired by Craig Venter.
  • The XPRIZE Board of Directors
  • The sponsors and donors of the prize.

All parties agreed with the need for and reasons behind the cancellation and approved the process of how we would communicate the decision to pre-registered teams and the public; the XPRIZE notified the two pre-registered teams–Ion Torrent and Wyss Institute at Harvard–of our decision prior to a public announcement on HuffingtonPost.com.

We understand one of the two pre-registered teams, the Wyss Institute at Harvard, disagreed with our decision and expressed concern that they were not consulted about our intentions to cancel the prize. In our discussions with them, XPRIZE agreed to add an important step to our process should we ever experience the unlikely need to cancel a prize: a public comment period (ratified by the Board of Directors).

Now, as part of our official competitor agreement, it states that: “XPRIZE will notify Teams of any anticipated cancellation of the Competition and will post a public notice of the same on the XPRIZE website. Included in this notice will be any available reasons for, and the proposed timing of, any anticipated cancellation. Team and the public will have at least thirty (30) days to comment on this notice prior to cancellation, and such comments will be taken into account by XPRIZE in their final decision.”

2. At the time of cancellation, was the Archon Genomics XPRIZE still relevant as an XPRIZE?

The Archon Genomics XPRIZE, launched in 2006, was the second XPRIZE competition. It was intended to inspire innovations in human genome sequencing with the potential to usher in a new era of personalized medicine. Teams in the Archon Genomics XPRIZE competition were competing for up to $10 million by becoming the first to rapidly and accurately sequence 100 whole human genomes to a standard never before achieved and at a cost of less than $1,000 per genome.

When we launched the Archon Genomics XPRIZE in October 2006, no one knew where the genomic sequencing market would go. In February 2001, Craig Venter’s groundbreaking approach had sequenced a single human genome for a total cost of ~$100,000,000 and took about one year. By 2006, it still took as many months to sequence a single human genome. The cost per genome was still more than $10 million, so it was clear that exponential improvements were necessary to radically transform the medical utility of genomics technologies.

From 2006 to 2011, the next generation genomics sequencing market (as measured in revenues) was growing at a compounded annual growth rate of 62% (~$1 billion split between instruments and reagents), which was much faster than we had anticipated. This rapid market growth was also driving a competitive environment that resulted in unremarkable product innovation–from 2008 through 2013, innovation outpaced “Moore’s Law” by many times. In 2013, the global genomic sequencing market reached sales of ~$1.4 billion (instruments and reagents)–a $10 million prize was no longer enough incentive, which dampened interest in the competition.

To account for this rapid and unprecedented market change, the XPRIZE had to re-launch the competition in 2011–entry barriers were minimized by establishing multiple prize purses to encourage more teams to register. This was done to keep the competition relevant, attempt to attract teams to compete and remain true to the intent of what an XPRIZE must accomplish: creating or catalyzing a new industry. However, despite these re-launch efforts, the genomics industry growth and the rate at which new technologies were being developed outside of the competition minimized the impact the prize outcome would have on the driving innovation in cost, accuracy, quality and speed of whole genome sequencing: At the time of cancellation, companies could sequence whole human genomes in a few days for around $1,000 per genome, and were nearly achieving the accuracy goal we set for prize.

The extremely low team interest we observed for this prize speaks to the difficulty the prize had to remain relevant in the market: At the time of the prize’s cancellation only two companies had pre-registered for the competition and none had entered into the formal Competitor Agreement required for participation. And only one team (Wyss Institute) expressed any concerns over this cancellation.

3. Would the market benefit in testing and celebrating progress in sequencing performance, speed and accuracy today? In the future?

XPRIZE believes that market forces have driven the quality, speed, and accuracy of whole human genome sequencing to the demanding edge of clinical application. Consumers of whole human genome sequencing will continue to benefit from competition arising from natural market forces to spur innovation towards even lower prices and clinical grade genomes. However, there still exists the need to improve the quality of whole human genome sequencing. Today the two key missing pieces to assess quality are: a reference genome and a widely accepted validation protocol that could be used to measure (or score) the accuracy and completeness of a technology platform against this reference genome. The XPRIZE believes that one important asset of the competition can help address one of these missing pieces: a validation protocol.

The Archon Genomics XPRIZE team worked with the global genomics industry to develop a new protocol for the analytical validation of whole human genome sequences. The first of its kind, this protocol was published through Nature Genetics and became the basis for new software capable of “scoring” whole genome sequences according to accuracy, completeness and haplotype phasing.

As our post-prize contribution to the industry and to celebrate the efforts the numerous thought leaders that contributed the Archon Genomics XPRIZE, we are planning to donate this novel validation software to the global genomics industry and integrate this platform into the National Center for Biotechnology Information (NCBI) GeT-RM Browser. Our hope is to promote widespread adoption of this validation software as a key tool in “scoring” a whole human genome against reference genomes under development by the National Institute for Standards and Technology (NIST) through the Genome-in-a-Bottle Consortium.

4. What should XPRIZE do to support the genomic sequencing industry and innovators going forward?

The XPRIZE can take some credit for inducing this new industry. When the prize was started analysts were arguing that the sequencing market was dead. The prize announcement changed the thinking of the field and stimulated its growth. As such, even this discontinued competition can and will have lasting benefits for humanity. The XPRIZE will strive to create a valuable legacy for the Archon Genomics XPRIZE with a critical post-prize initiative: open-sourced access to the immortalized cell lines AND the fully sequenced genomes of the individuals who donated their DNA to support groundbreaking longevity research.

The Archon Genomics XPRIZE competition gathered DNA samples from more than 100 centenarians from around the world. This DNA was to have been used during the competition to assess teams capabilities of quickly, cheaply and accurately sequencing whole human genomes.

We plan to have these centenarian genomes fully sequenced and to provide the resulting genomic data to researchers in an open forum. We believe that the global research community can use this information to support groundbreaking longevity research by learning what enables these individuals to live long lives and evade disease.

XPRIZE welcomes your comments and suggestions. Please send inquiries via email to: [email protected].

George Church, Professor, Harvard, MIT, Wyss & Broad Institutes

Opening Statement

First, I thank Peter Diamandis, Marc Hodosh, Craig Venter, Larry Kedes, Grant Campany, the Blussons, etc. guiding the Archon Genomics XPRIZE (AGXP) since Oct-2006 -- also teams at LifeTech, Kollmorgen, Nikon, Illumina, Complete Genomics (CGI) and NIH-NHGRI with whom we have worked toward this goal. I’m thankful for this opportunity to address, and possibly reverse, an unfortunate $10M decision. Comments below are aimed at future incentives, not to question the good intentions of all involved in AGXP.

My team registered in Apr-2008 (with the Prize stated to go to PersonalGenomes.org) and Ion Torrent officially registered in Jul-2012 for AGXP. By Fall 2013, the XPRIZE foundation had spent $6M preparing and had over $10M available for Prizes. All that was left to do was the thirty day evaluation of our efforts, beginning 5-Sep-2013. Three top technologies (those of Life Technologies, Illumina and CGI, which we had aided) – were far ahead of other technolgies for accurate human genome sequencing. One AGXP team was Life Tech and the other team, which I headed at the Wyss Institute, was working closely with Illumina and CGI. So it would have been (and could still be) a ‘match race’ involving the best technologies.

1. What should be the procedures and circumstances for cancellation of an XPRIZE?

Appropriate circumstances for cancellation might include all contestants withdrawing or discovery of a physical impossibility. Appropriate procedures for cancellation should include a ‘reality check’ with all registered teams well in advance of announcement, to see if the teams agree that the reasons for cancellation are valid. If there is only one contestant, then that team must still meet all criteria of the contest to win. This is similar to what happens in asynchronous challenges such as thatof the Ansari XPRIZE, in which only one team might be competing at a given time.

Inappropriate reasons for cancellation would include two cases:

A. The field seems to be has advanced too slowly or seems to be unresponsive to the XPRIZE. This is an inappropriate reason, since they may be further along than they seem or the dificulties could be informtive. The fact that the teams are willing to invest in technology to meet the competition criteria should be adequate for continuing with the competition. Such an investment could lead to later valuable insights from the competing teams or the various observers. This is what happened in the first round of the DARPA driverless car challenge.

B. The field has advanced at exactly the speed required to achieve the goals of the prize, such that all teams could win or tie. If the field has developed additional ‘independent’ incentives since the start of competition, that represents success, not cause for cancellation. It is not a failure if “market failures that prevent innovation
are being solved by current industry developments”. If the goal of the prize is to alter the industry, then the teams should not be penalized if they help to alter the industry. In case B, each team must still complete all tasks and then on completion, the prize distributed equally. Even if there were overwhelming evidence (there wasn’t such evidence) that the time chosen by the XPRIZE for the evaluation was imperfect, the contestants should not be penalized for this – and not let a notion of “perfect” competition interfere with an otherwise excellent learning moment (and fair follow-thorough on an agreed goal).

AGXP says that “technology performance issues were being solved at a rate that outpaced all technological and economic expectations.” In actuality the performance issues were being solved at exactly the rate needed to deliver meet the specifications of the XPRIZE by the deadline set.

AGXP has said “By 2013, the prize had neither officially registered teams nor any Competitor Agreement signed with teams”. However, this contradicts the AGXP public announcement on 23-Jul-2012 “Ion Torrent Officially Registers” (goo.gl/HhX6Cy). My team had registered in AGXP (Apr-2008, goo.gl/01LEHM), and this was re-affirmed in an AGXP announcement 3-Oct-2012. Both teams had paid registration fees and were encouraged to spend significant resources to achieve the goals of the AGXP.

2. At the time of cancellation, was the Archon Genomics XPRIZE still relevant as an XPRIZE?

Yes. The prize would (then and now) help inspire, educate and evaluate progress. It was (and still is) far from obvious that any team could achieve the “Best-in-Class requirements” of $1K each for 100 genomes in 30 days, 98% complete, 100% haplotyped, and less than one error per Mbp – or even achieve the “minimum requirements” of $10K each, 95% complete and less than one error in 100,000 bases.

AGXP says “By 2006
The cost per genome was still more than $10 million.” Actually, by 2006 there was not a single published example of a diploid human genome sequence, despite $3 billion spent by several companies and academic factories. The AGXP says “In 2013, the global genomic sequencing market reached sales of ~$1.4 billion (instruments and reagents)–a $10 million prize was no longer enough incentive”. There are $billion markets behind nearly all of the XPRIZEs. Very few of those funds are set aside for R&D, and an even tinier subset for truly disruptive goals. What is the evidence that $10M is not a huge incentive to key individuals in those teams?

I have been collaborating with Complete Genomics (CGI) and Illumina/Solexa/Lynx (since 2006 and 2000, respectively) and our publicly stated strategy would be a mixture of those methods and other technologies that we had not yet exported to any company. The rules permit use of any commercial reagents or devices or collaborative teams. The XPRIZE staff asked me early on to enable representation of CGI and Illumina technologies in AGXP. A combination of these would be particularly powerful.

The AGXP says “To account for this rapid and unprecedented market change, the XPRIZE had to re-launch the competition in 2011.” It is not evident that AGXP “had to” shift the goal posts in 2011 (or in 2013). The impact of following through would have been far more positive than unilaterally pre-judging the event. It is relevant to reflect on the story of King George III intervening in 1772 on behalf of watchmaker John Harrison for the Longitude Prize after 40 years of work and 11 years of Prize committee stonewalling. (wikipedia.org/wiki/John_Harrison).

3. Would the market benefit in testing and celebrating progress in sequencing performance, speed and accuracy today? In the future?

Short answer: Yes and Yes.

“What we realized is that genome sequencing technology is plummeting in cost and increasing in speed independent of our competition. Today, companies can do this for less than $5,000 per genome, in a few days or less – and are moving quickly towards the goals we set for the prize. For this reason, we have decided to cancel an XPRIZE for the first time ever.” -- Huffington Post 22-Aug-2013 Peter Diamandis (the official ‘explanation’ of cancellation).

Is this a reasonable prize policy? – to cancel if there is too much progress -- at the last minute -- after teams have spent time and funds to prepare? Isn't “moving quickly towards the goals” exactly what every winning team has done (and should do) in past (and future) XPRIZE competitions? If the point of the X-prize is to encourage passionate inventors to compete with all their heart, holding nothing back, then why cancel? The AGXP could have been (and could still be) a high profile event with very positive outcomes for everyone, instead of an avoidable slap at innovators. Six factors of ten improvement in seven years is unprecedented, and deserves exuberant celebration — not cancellation. The AGXP says “An XPRIZE cannot be awarded to honor past achievements or celebrate an accomplishment that could otherwise occur without the prize.” It is conjecture that the genomics accomplishment would have occurred without the AGXP incentive (to any extent different from other XPRIZEs). Indeed companies often try to prevent commoditization and precipitous price drops. When a competition runs from 2006 to 2013, by definition, it will include recognition of “past achievements”. That is quite acceptable.

A small number of visionary team leaders, with whom I have had the great pleasure to collaborate (Jonathan Rothberg, Greg Lucier and Barry Merriman at Life Tech; John West, Jay Flatley and Mostafa Ronaghi at Solexa/Illumina; Kevin McCarthy at Kollmorgen; Rade Drmanac and Cliff Reid at CGI) have been striving for the goals of the AGXP. The goals are not yet met and the innovators’ causes could still benefit from the $10M. The XPRIZE was (and could still be) a good way to incentivize and evaluate progress on vitally needed genome sequence quality.

The XPRIZE “believes that one important asset of the competition can help address one of these missing pieces: a validation protocol.” One way to test this protocol would be to finish the AGXP competition (not cancel it). The AGXP mentions the idea of “reference genome” and the NIST + FDA “Genome-in-a-Bottle Consortium”. Note that the reference genomes chosen for this project were from PersonalGenomes.org. This valuable non-profit would have benefitted the entire genomics community via the $10M prize donated by our AGXP team (if we won).

4. What should XPRIZE do to support the genomic sequencing industry and innovators going forward?

XPRIZE.org should restart the genomic XPRIZE (as soon as possible) with the same rules, but this time with the teams protected from cancellation by employing the above considerations (in answer #1, above). It should endeavor to regain the trust of the innovators who are carefully watching the outcome of this discussion about restarting the AGXP.

The AGXP, should it recur, could offer a prize for even higher accuracy and integration with interpretation. An existing infrastructure for this exists in the form of CAGI (Critical Assessment of Genome Interpretation, genomeinterpretation.org). That organization would welcome collaboration with XPRIZE (especially if the reputation for inadequately justified cancellation is reversed). There should also be support for efforts to increase genome literacy (such as pged.org).

The reinstated GXP could more directly champion (as I have) truly new goals in genomics technologies that could be transformative, for example, single cell and subcellular in situ sequencing, gapless de novo sequencing, 3D-genome structure, somatic variation, and integrating sequencing with genome engineering (e.g. with CRISPR or MAGE) to determine causality of variants of unknown significance.

In addition to testing the technology, the AGXP was to have sequenced 100 Centenarian genomes. Similarly, in addition to the technology, my group has had keen efforts in aging research for a decade. Over 3 million people die every month from causes that are rare in youth. If the 100 AGXP genomes were to result in breakthroughs that help to extend some of those 3 million lives, then delay in making those data public from 5-Oct-2013 to 1-April-2014, would mean failure to prevent over 10 million deaths. (Note, this is meant poetically, not legally). Obtaining very high quality genome sequences may be crucial -- especially, defining inversions, translocations and haplotypes which are hard by conventional (i.e. pre-AGXP) methods. Those 100 genomes are ready, as are the two AGXP teams. So we should begin the 30 day sequencing of these 100 genomes as soon as possible.

The $10M prize money would be donated (as planned since 2008) to the non-profit PersonalGenomes.org (the world’s main open-access resource for human genome + trait data ), which would be used to sequence and share another 10,000 human genomes. Such genomes (+ traits) would have huge impact in a community otherwise unable or unwilling to share such valuable biomedical information. Anyone reading this essay could now vote for either of two outcomes: A. continued cancellation of AXGP -or- B. restarting AGXP to support and test the efforts of the AGXP teams. You can vote via your favorite social media.

Share this page

Related News

Become a Part of the Future! Sign Up for Our Newsletter: