2012年12月13日木曜日

OBAMA Project Norwhal Work Continued

オバマの選挙戦解析が報道された。

2008年 「情報を配信する線」として活用。
2012年 「情報収集・解析・予測・絞り込み」に活用。

資金集め
・過去の支持者データベースや外部データを活用し、寄付する可能性の
 高い人のリストを作り、寄付を呼びかけたり、対象の年令・性別・
 土地柄に合わせて資金集めパーティの目玉セレブを選択。

投票の呼びかけ
・シミュレーションに基づき、それぞれのターゲットに合わせた発信者
 と内容のメッセージが送られた。

Project Norwhal
・人件費まで含めたキャンペーン中のITへの支出額は、オバマ陣営は
 ロムニー陣営の2/3。
 前回の選挙以来バラバラだった数多くの支持者リストを統合し、外部
 のデータも活用し、膨大なデータを扱い膨大な回数の演算をしながら、
 無料のオープンソース・ソフトを徹底的に使い倒してコストは最小限。
・Narwhalが成功すれば、、オンライン活動家、オフラインの投票者、
 寄付金提供者、ボランティアらを融合する可能性がある。
  1億7000万人の潜在的投票者
  1300万人のオンライン・サポーター
  300万人のキャンペーン寄付提供者
・web登録者の情報はBlue State Digitalが管理。
  100人のボランティアが投票者情報を登録。
  支持者への支持依頼の省略。
 キャンペーン基金調達チームは、提供者のリストを管理。
 フィールド ・チームは、Build the Hopeなるボランティアデータ
 ベースを管理。
 Akamai、ChartbeatとGoogle Analyticsを利用。
・Facebookアプリを導入し、潜在的オバマ支持者をオバマに投票する
 ようにした。
・特定の政策を支持する人のみに、特定の政策に関する情報を提供。
・サーバOSは、Linux Ubuntuを採用。
・インフラは、アマゾン・ウェブ・サービス(AWS)。
・チームがつくったソフトウェアは99.99%AWSでホストされ、コストを
 下げサーバー管理の手間を省いた。
・アプリケーション数は200以上。
 10程度のDBMS/NoSQL systemsとPython、Ruby、PHP、Java、Node.js言語
 で作成。SOAを採用し、データ解析処理を含む。
 同時に利用するユーザー数は7000人、投票日直前に電話件数が急激に
 増え、最後の4日間だけで200万本の電話をかけたが、このためのツール
 がAWSでホストされた。
・アマゾンへの支払額は$257,287.97。
 選挙の直前にトラフィックが極端に上がり、終わったらほとんどを
 撤去という使い方に最適。
・S3(Simple Storage Service)
 クラウド・サービスから成るAWSの一つの構成要素。
 サーバー障害に備えたバックアップにも利用。
 ウェブサイトのスナップショットを定期的にとり、S3に保存。
 障害があれば最新のスナップショットにリダイレクト。

オバマ選挙戦技術チームは、個人情報による投票動向分析処理をしている
と思っていたが、投票者の囲い込みもしていたようだ。

地域ボランティアが、支持依頼で個別訪問している際に、訪問宅の人種、
家族構成等を収集し、報告していた。データベースに登録して、利用する
のだろうが、支持者が多ければ、より詳細な分析ができるが、支持者が
少なければ、大雑把な分析になる欠点が考えられる。今回の選挙では、
効果的だったようだが、特定の条件下でしか運用できない可能性がある。

Project Norwhalの運用開始が、選挙運動開始日となれば、1年以上前から
準備されたはずだが、資金は、前回の選挙でのキャンペーン基金を充てた
のだろうか。未だに、選挙キャンペーン基金のその後は不明なまま。

オバマが退任後、Project Norwhalは民主党が引き継ぐのか、市場で売ら
れるのだろうか。

話題にならないオバマの暗闇
Big Data Vote Review
米大統領選 勝敗分析


---【海部美知 for Cloudian】AWSを使ったオバマのケチケチ「マネーボール」作戦を企業も活用へ---
クラウディアン株式会社 2012年12月08日
エノテック・コンサルティング代表
海部美知
http://japan.zdnet.com/company/topic/30000318/

◆ 大統領選を制したビッグデータ
 11月のアメリカ大統領選では、民主党オバマ大統領側がITとネットによる情報収集力をフル活用し、接戦と伝えられた選挙戦を制する原動力となったともっぱらの評判だ。
 2008年の選挙の時は、どちらかというと「有権者にリーチする新しい手段」として、フェースブック・ツイッター・ユーチューブ・スマートフォンなどを、「情報を配信する線」として活用することが中心だった。これに対し、今回の選挙ではソーシャルによる情報配信はもはや当たり前で、それよりも「情報収集・解析・予測・絞り込み」、つまり「線」ではなく「脳」の部分にビッグデータ手法を活用し、もっと高度な使い方をした。
・資金集め: 過去の支持者データベースや外部データを活用し、寄付する可能性の高い人のリストを作り、寄付を呼びかけたり、対象の年令・性別・土地柄に合わせて資金集めパーティの目玉セレブを選択し、カリフォルニアではジョージ・クルーニーを招いて大成功を収めたなどが知られる。
・投票の呼びかけ: 膨大なシミュレーションに基づき、それぞれのターゲットに合わせた発信者と内容のメッセージが送られた。たまたま私の手元には、ミシェル・オバマ夫人から「投票に行きましょう」というツイッターのダイレクトメッセージ が届いた。ちなみに対抗するロムニー候補からは、2500ドルの寄付を依頼する紙の手紙が来た。ロムニー氏の場合は、おそらく、大学の名簿から私の名前と住所を引っ張ったと思われるが、この対照に思わず笑ってしまった。
・テレビCMや遊説: 伝統的な選挙運動についても、どの場所でどのような演説をするか、支持してくれそうな視聴者層にリーチするにはどの番組にどんな広告を出すか、といったことも細かくデータ解析ではじき出した。
◆ ケチケチ勝つ「マネーボール」作戦を支えたAWS
 こうした戦法は、「データ解析を駆使して小が大に勝つ」の代名詞となった例の映画にちなんで「マネーボール」作戦ともよばれる。
 ARS Technicas誌記事 (How Team Obama's tech efficiency left Romney IT in dust) によると、人件費まで含めたキャンペーン中のITへの支出額は、オバマ陣営はロムニー陣営の2/3に過ぎなかったという。オバマ陣営は、前回の選挙以来バラバラだった数多くの支持者リストを統合し、外部のデータも活用し、膨大なデータを扱い膨大な回数の演算をしながらも、無料のオープンソース・ソフトを徹底的に使い倒してコストは最小限に抑えた。
 そんなケチケチ作戦を支えたインフラが、アマゾン・ウェブ・サービス(AWS)だった。チームがつくったソフトウェアは99.99%AWSでホストされ、コストを下げサーバー管理の手間を省いた。アマゾンにホストされたアプリケーションの数は200以上、しかしアマゾンへの支払額は$257,287.97で、大規模な運用の割に少ない。(キャンペーン中の経費明細が記事の中で詳細に公開 (How Team Obama's tech efficiency left Romney IT in dust) されている。)そして、選挙の直前にトラフィックが極端に上がり、終わったらほとんどを撤去するという使い方にAWSは最適、というよりAWSがなければほぼ不可能とすら言えるだろう。
 AWSに関するブログ (How Team Obama's tech efficiency left Romney IT in dust) では、キャンペーン中に支援依頼の電話をかけるための管理ツールの例が示されている。同時に利用するユーザー数は7000人、投票日直前に電話件数が急激に増え、最後の4日間だけで200万本の電話をかけたが、このためのツールがAWSでホストされた。この間突然データ量が増え、選挙が終わればゼロとなる。
 S3(Simple Storage Service)は、さまざまなクラウド・サービスから成るAWSの一つの構成要素であり、サーバー障害に備えたバックアップにも利用された。ウェブサイトのスナップショットを定期的にとり、S3に保存しておいて、障害があれば最新のスナップショットにリダイレクトされる。投票日の前にハリケーン・サンディが東海岸を襲ったのは、ちょうどよい予行演習になったとのことだ。

◆ 企業にも活用されるAWS・S3
 このように、アマゾンAWSおよびS3は、ウェブサービスだけでなく、政府や企業にも使われるようになっている。
 オバマ選挙運動のビッグデータ・チームは、11月末に開催されたアマゾンの開発者会議re:Inventに登場した。私は参加できなかったが、見るからにナントカな人々 である。
 このre:Inventでの講演で、ポリシー・ベースのアーカイブサービスが紹介されている。企業の基幹系システムの場合、アクセス頻度の高い「Tier1」データは高速でアクセスできるSANに保存、その後一定の期間が過ぎるなど、なんらかのポリシーに基づき「Tier2」に移され、さらにその後ディスクやテープに落として倉庫に保管となる。高速アクセスのできるストレージほど高価だからだ。これが最近では、Tier2にS3を利用し、さらに数時間後に取り出せればよいという、ディスク・テープの代わりとなるGlacierというサービスも提供されている。
 re: Inventは今回が初回であったが、ZDNet (Amazon Web Services cuts S3 prices, knocks old guard rivals) によると、参加者6000人という盛況だったようだ。この会議での報告によると、現在AWSは190地域に数千の顧客を持ち、S3のオブジェクト数は今年6月の1兆個からすでに1.3兆個に達し、一秒あたり83.5万件のリクエストを処理し、370万のクラスタを持つという。2011年に81件であった新サービス・機能は2012年では158件に増加、一日に50億ドル級の世界企業一社分の容量を増やし続けている。
 一方で、今年にはいってすでに23回の値下げをしてきたが、今回24回目の値下げを発表した。グーグルとの競争などがその背景にある。
 こうしたパワーに支えられ、S3は高い堅牢性と無制限の容量を提供し、政府や企業のITシステムにも浸透しつつある。従来、企業の基幹システムでは、安全性への懸念などから、クラウドの利用は必ずしも進んでいなかったが、最近は用途により、パブリック・クラウドやプライベート・クラウドを使い分けるメリットが認識されてきた。
 オバマのマネーボール作戦のように、クラウドに適した用途にうまく安価なパブリック・クラウドを活用することで、企業のサーバーやストレージのコストを大幅に下げ、競争に勝ち抜き、より高いマージンを確保することが、今後ますます重要になっていくだろう。


---オバマ氏再選の裏にアマゾンの力――2012年米大統領選を支えたクラウド基盤---
2012年11月17日
Brandon Butler/Network World米国版
http://www.computerworld.jp/topics/601/205544

選挙期間限定の“一時的”なIT基盤を構築、選挙終了後はほぼすべてを解体

 バラク・オバマ(Barack Obama)米国大統領再選の裏側には、管理ツールや分析ツールなど200種類以上にも及ぶ各種カスタム・アプリケーションがあった。そうしたITサービスの広範さや複雑さは、巨大企業における運用規模にも匹敵する。その選挙期間限定の“一時的”なIT基盤を提供したのが、業界最大手である米国Amazon.comのクラウド・プラットフォームだ。
 とは言うものの、基本的にほぼすべてのIT基盤は選挙終了後に“解体”されてしまった。これぞ“クラウドの力”である。「“ミッション・クリティカル”とは、まさにこうした状況のことだ」とAmazon.com傘下であるAmazon Web Services(AWS)のエバンジェリスト、ジェフ・バー(Jeff Barr)氏はオバマ陣営におけるAWSクラウドの活用状況を思い起こす。「米国を先導するという貴重な機会を得られるものの、リスクは大きかった」とバー氏は述べる。
 AWSによると、オバマ陣営では同社クラウドで利用できるほぼすべてのサービスを活用し、投票日が近づくにつれてサービスを急速にスケールアップしていったという。
 例えば、投票者の登録管理では、「Amazon Relational Database Service(Amazon RDS)」上にデータベースを構築した。また、「Amazon Elastic Compute Cloud(EC2)」上に構築した分析ツールは、対象投票者に関する動的な情報をリアルタイムに提供したり、潜在投票者を細分化したりする機能、またリアルタイムなフィードバックに基づいてどこにマーケティング資金を投入すれば効果的であるかをリコメンドする機能などを提供した。
 AWSの報告によると、支持を求める電話作戦用のツールは、7,000ユーザーの同時利用を可能とし、選挙運動の最後4日間における電話件数は200万件に及んだ。「今回の選挙運動ではAWSを利用することで、通常であれば数千万ドルかかるIT投資を抑えた」とバー氏は書いている。
 選挙が終わった後、選挙活動で取得した情報はすべて「Amazon Simple Storage Service(Amazon S3)」上にバックアップし、「(IT基盤を)大幅に縮小した」という。
 AWSは11月末にラスベガスで同社初のユーザー・イベントを開催する予定で、オバマ陣営の関係者が今回の選挙活動の裏側について講演することになっている。また、同陣営の最高技術責任者(CTO)、ハーパー・リード(Harper Reed)氏も、「Big Data and the US Presidential Campaign(ビッグ・データと米国大統領選挙)」と題したパネル・ディスカッションに登壇する予定となっている。
 一方で、共和党の挑戦者、ミット・ロムニー(Mitt Romney)氏が有する分析プラットフォーム「Orca」は投票日当日に大規模な障害に見舞われたとする報道も最近話題となっている。


---What Barack Obama can teach Tesco and co---
Wednesday 05 December 2012
Mark Piesing
http://www.independent.co.uk/life-style/gadgets-and-tech/features/what-barack-obama-can-teach-tesco-and-co-8386845.html

Will Obama’s data-mining 2012 campaign offer retailers ideas for identifying new customers? Mark Piesing reports

The friendly big red capital letters of the email welcoming me to the world of online gambling - “WELCOME ABOARD. GAME ON” - were in marked contrast to the Orwellian feel of the site’s age-verification policy that I had found just a click away.

It promised to use a third party to electronically verify that I was who I said I was by using my name, address, date of birth and phone number, and said the results might be retained for use by other businesses in the future. It meant that whatever I shared and whatever they had on me was going to remain forever out there in cyberspace, beyond my control.

And if I’d failed the test, I may even have had to verify my identity through a Skype interview.

According to John Aristotle Phillips, speaking at this year’s Silicon Valley Comes to Oxford event, this experience is likely to become more common as mainstream brands such as Tesco seek to follow the lead set by the “sin” industries of gambling, drinking and tobacco in applying one of the main lessons of Barack Obama’s victory last month - the effectiveness of big data.

“Big data” is a catch-all term for the collection, analysis and use of very large and complex sets of data. By asking people to volunteer their personal data, and then combining it with publicly available data, including what they post on social networks, Obama’s team was able to build a profile of individuals and their habits, made up of  50 or so individual data points, and target them more effectively.

So good were they at this nano-targeting that as one analyst put it, “the Obama campaign was able to identify voters that Romney’s camp didn’t even know existed”. “While some companies have been doing this effectively, many haven’t,” Phillips says.

Phillips is co-founder of the market-leading  political technological consultancy Aristotle Inc, and has advised every US president since Reagan  plus a  few British prime ministers. His interest in the value of publicly available data began as an undergraduate at Princeton, when he used documents in the library to design an atomic bomb that the Pakistanis promptly tried to buy - earning himself the label “the A-bomb kid” in the process. Aristotle provides its integrity identity verification software to “seven out of 10 UK online gambling sites, and it’s soon to be eight”.

“I have always been surprised by how many businesses in the UK let visitors leave their sites without giving them much in the way of personal information,“ says Phillips. “There has been a misconception that if you ask people who they are they will run away from your site. Now though more companies are embracing identify verification.” Yet it is questionable how much people understand what this data could be used for.

For Phillips, what has made business “sit up and take notice” of  the Obama campaign was the effective way they used a centralised database called Narwhal. It collected all the data the campaign had on an individual to build in effect a one-to-one relationship with the voter, enabling them to make decisions about the conduct of the campaign on a street-by-street basis.

Through its Facebook app, the Obama campaign asked voters to share their Facebook friends with the  campaign so that potential Obama voters could be identified. According to recent figures from the campaign, 20 per cent of targeted friends  responded in the “correct way”.

Charles Duhigg is a New York Times journalist and author of The Power of Habit. Like Phillips, he believes that “lots of people gave up their data during the election because all Obama was trying to do was to get his vote out. Whether they knew this meant they were going to get targeted - or would have cared if they did - is much harder to tell.”

Unlike Phillips, however, Duhigg believes that “this is going to be much harder to do for someone like Tesco, as people don’t tend to share with companies as they do with parties”. It is also going to be harder for any business to reproduce the effect in Europe, “as the truth is, in the US we just don’t get as worked up by privacy as the Europeans do”.

Daniel Kreiss, author of Taking Our Country Back and assistant professor at the School of Journalism and Mass Communication, University of North Carolina at Chapel Hill, is also sceptical as to whether people will hand over data in the same way to high-street brand names. For him, it is more a question of whether businesses in the UK will be able to use data to the same degree of effectiveness as Obama’s campaign did.

Phillips disagrees. “I think it’s the other way around, and I’m in the political business. I think most people tune in only during election season. Rest of the year, they want to be able to do the things that identity verification will  facilitate in their day-to-day lives.”

However, for Jorn Lyseggen,  founder and CEO of the global software company Meltwater Group, “what matters for business is not so much any particular technology or algorithm Obama’s team used, what matters is what they did worked”.

“Obama’s effective use of data mining has been a real proof point for the power of the concept of big data, as its accuracy astonished the world and baffled scientists, businessmen and analysts.”

“Now the penny has dropped and business executives are going to take the insights big data offers into their company seriously, as they can clearly see what a rather difficult concept actually means in concrete terms.”

Like Phillips, Lyseggen believes that up to now there has been a lot of uncertainty as people have been asking how it can create value for them. He thinks the success of the Obama campaign presents business with an “extraordinary opportunity” to be able to analyse in real time their customers’ attitudes to them and to their competitors.

For Duhigg and Kreiss it marks the death of intuitive leadership in business. No one is going to follow their gut and disagree with the data again.

Despite the feeling that this is something Orwell dreamt up, Phillips thinks the spread of identity verification comes down to Darwin. Business leaders in the sin industries and politicians in particular “adapt quickly to what works and compete very hard”.

“While they want what works in 2012, they are going to want to know what wins in 2014 or 2016.” And the rest of the world will follow. Lucky us?


---Check out how Obama saved $14.5 mn through open source---
Published On :Tue, Nov 27,2012
http://www.ciol.com/ciol/news/122377/obama-saved-usd145-mn-source

"Four more years. This happened because of you. Thank you," Obama tweeted soon after he defeated his Republican rival Mitt Romney in a closely contested 2012 US presidential poll.

Well, we are aware of the fact that the President of the United States of America and his tech team were all over the Internet embracing different kind of tools -may be from social media or from different online campaigns - to win the 2012 presidential elections, but many of us are not aware that open source software also played an important role during the US elections.

Yes, according to arstechnica.com, President Barack Obama's campaign underspent Romney's on IT products and services by $14.5 million, putting the money instead into building an internal tech team.

The report in Ars tech says, "The Obama campaign, all-inclusive, spent $9.3 million on technology services and consulting and under $2 million on internal technology-related payroll, while Romney campaign spent $23.6 million on outside technology services-most of it on outside "digital media" consulting and data management."

The report further added, the bottom line is that the Obama campaign's emphasis on people over capital and use of open source tools to develop and operate its sophisticated cloud-based infrastructure ended up actually saving the campaign money.

"Key in maximizing the value of the Obama campaign's IT spending was its use of open source tools and open architecture. Linux-particularly Ubuntu-was used as the server operating system of choice. "We were technology agnostic, and used the right technology for the right purpose," VanDenPlas, lead DevOps for Obama for America was quoted as saying by arstechnica.com.

"Someone counted nearly 10 distinct DBMS/NoSQL systems, and we wrote something like 200 apps in Python, Ruby, PHP, Java, and Node.js."

This is not the first time where US has embraced on open source technology.

It may be noted that in July 2009, Rock the Vote, a provider in technology solutions for election process in US, had announced that it has partnered with Open Source Digital Voting (OSDV) Foundation's TrustTheVote Project, to introduce new online voter registration tools based on open source technology.

So, Do you think open source also plays an important role as equivalent to proprietary software? Let us know your feed back...!


---Obama's Campaign Technology: Lessons in Big Data Analytics---
Added by Karen Hanna on Nov 26, 2012
http://midsizeinsider.com/en-us/article/obamas-campaign-technology-lessons-in

Obama's campaign technology may have changed the game for future political campaigns. Instead of relying solely upon the track records of seasoned consultants to lay out strategy based on gut instinct, experience, and interpretation of the poll-of-the-day, Obama's strategy was to take a calculated leap toward data-driven campaigning by creating a system of several tools and analytical models to better understand the voting populace, ultimately using big data to characterize voters down to individuals. What may be surprising for IT analysts at midsize companies is that the tools used by the Obama team to develop their architecture, dubbed Narwhal, are not that exotic. According to an Ars Technica article, the technology team used the cloud, Amazon Web Service, and open source software. By the New York Times' count, about 200 applications were written, including the data analytics programs.

For midsize companies, particularly those that are evaluating their big data analytics needs, the Obama campaign example holds several lessons related to the challenges of getting useful information out of big data, and in developing complex analytical systems in a relatively short amount of time.

One big lesson stems from the 2008 campaign, when the Obama team lacked its own internal IT team. As a result, field offices built their own tools, none of which shared information with any other system. For 2012, the Obama team knew it had to build a common system of tools. Further, the applications they built had to scale up.

Midsize companies should be able to relate; they may have big data to exploit but lack internal IT resources to design and develop a common solution. This may result in a patchwork of systems built across departments that don't scale and don't share data.

For their solution, the Obama technology team built Narwhal using services oriented architecture (SOA) principles. An application programming interface (API) allowed the various applications to be built using different languages but still have the ability to share information across a common data store.

There is a sensibility in building these "right-sized" systems that have the ability to scale up as a company's needs and accompanying big data growth and to have these systems built so that information can be shared across systems. It is worthwhile for IT at midsize businesses to keep this in mind as they design their own systems. Architecture is important; it may be feasible to hire consultants to assist a small internal team as they build out a large scale system, or IT may find it feasible to incrementally build systems over time. No matter which approach is taken, it must have a firm foundation on which to build.

Narwhal 1 1995-06-10Another interesting aspect of the technology used by the Obama team is that of internal social networking. Some of the applications developed were more widely accepted and used because of the applications' social networking capability, allowing volunteers to connect, share their experiences, and build business relationships. This isn't unlike the benefits seen by midsize companies that use commercially-developed enterprise social networking solutions.

The presidential election was a unique example of a hard IT deadline that had to be met, and Obama's campaign technology strategy was ultimately successful. According to an article in the Los Angeles Times, in spite of the apparent disbelief of some news pundits, the Obama analysts in Chicago "were watching state after state fall exactly as they had predicted." Perhaps, the biggest lesson to be gleaned from this election is that IT can truly design, plan, build a system of systems and adapt, even with a finite project deadline, provided of course that the architecture is well thought out and firmly established at the outset.


---Obama's tech team used open source and some Microsoft software to fuel its victory---
By Preston Gralla
November 21, 2012 1:40 PM EST
http://blogs.computerworld.com/windows/21376/obamas-tech-team-used-open-source-and-some-microsoft-software-fuel-its-victory

The presidential campaign wasn't just a battle of political philosophies -- it was also a duel between the two campaigns' IT strategies and tools. Ars technica reports that Obama's team was far more effective than Romney's, and used plenty of open source tools...and some software from Microsoft as well.

The article provides an in-depth look at the IT strategy of the Obama campaign. Ars technica also examined the Romney campaign's IT strategies as well. The differences between the campaigns were stark. Obama's campaign spent far more money building its own IT staff than did Romney's, and used open source and cloud tools to quickly develop apps and technology when needed. Romney's team, by way of contrast, spent much less on internal staff, but far more on consultants.

According to the ars technica article, the Obama campaign spent under $2 million on internal IT payroll, and $9.3 million on technology services and consulting. Romney's campaign, meanwhile, spent $23.6 million on outside technology services. Overall, the article reports, Obama's campaign spent $14.5 million less on IT products and services -- both internal and external -- than did the Romney campaign.

Despite that, Obama's team and technology far outperformed Romney's. For example, Romney's big-data Orca app for turning out the vote was an epic failure.

ars technica notes that Obama's team heavily relied on open source tools:

    Key in maximizing the value of the Obama campaign's IT spending was its use of open source tools and open architectures. Linux-particularly Ubuntu-was used as the server operating system of choice. "We were technology agnostic, and used the right technology for the right purpose," [Scott VanDenPlas, lead DevOps for Obama for America] said. "Someone counted nearly 10 distinct DBMS/NoSQL systems, and we wrote something like 200 apps in Python, Ruby, PHP, Java, and Node.js."

Even though the campaign relied on open source, it was also a buyer of Microsoft technology, spending $522,210.04 for software licenses, at about $500 per staff member, reports art technica.

In a sense, the campaigns' differing IT strategies mirrored their different political philosophies. Romney essentially outsourced everything (just as he favors outsourcing many governmental functions), while Obama's campaign relied more heavily on his staff (just as he favors relying on government itself to perform many vital services). In this case, at least, in-house expertise trumped outsourcing, and proved to be more agile and flexible as well.


---How Team Obama's tech efficiency left Romney IT in dust---
by Sean Gallagher - Nov 21 2012, 9:00am JST
http://arstechnica.com/information-technology/2012/11/how-team-obamas-tech-efficiency-left-romney-it-in-dust/

Obama campaign's tech team beat Romney by using opposite strategy-"insourcing."
Despite running a campaign with about twice the money and twice the staff of Governor Mitt Romney's presidential bid, President Barack Obama's campaign under-spent Romney's on IT products and services by $14.5 million, putting the money instead into building an internal tech team. Based on an Ars analysis of Federal Election Commission filings, the Obama campaign, all-inclusive, spent $9.3 million on technology services and consulting and under $2 million on internal technology-related payroll.

The bottom line is that the Obama campaign's emphasis on people over capital and use of open-source tools to develop and operate its sophisticated cloud-based infrastructure ended up actually saving the campaign money. As Scott VanDenPlas, lead DevOps for Obama for America put it in an e-mail interview with Ars, "A lesson which we took to heart from 2008 [was that] operational efficiency is an enormous strategic advantage."

As we revealed in our recent analysis of the Romney team's tech strategy, the Romney campaign spent $23.6 million on outside technology services-most of it on outside "digital media" consulting and data management. It outsourced most of its basic IT operations, while the Obama campaign did the opposite-buying hardware and software licenses, and hiring its own IT department. Just how much emphasis the Obama campaign put on IT is demonstrated by the fact that the campaign's most highly paid staff member was its Chief Integration and Innovation Officer, Michael Slaby, with an annualized salary of about $130,000.

By comparison, Kevin Rekowski, the Romney campaign's Director of Technology, was barely in the top 20 salaries of the Romney campaign, with an annualized salary of $80,000. Zac Moffatt, Romney's Digital Director -a social media planner, not a technology expert-was number five, at $175,000 a year, in addition to whatever he earned from hiring his own firm, Targeted Victory, to handle much of the Romney campaign's digital strategy.

But the advantage of having a personal army of coders wasn't just financial. "Campaigns are serious tests of your creativity and foresight," VanDenPlas explained. "They are unpredictable, agile, and short-an 18 month, $1 billion, essentially disposable organization. Hackers can thrive in an environment like that, to a point where I'm not sure anyone else really can. Everything is over far too quickly to get boring."
Smart, not perfect

The strategy the Obama campaign's DevOps team used to manage the ever-growing number of applications deployed by the campaign was to "choose the lowest cost route to get us the most results-basically, be smart, not perfect," VanDenPlas said. "We did a lot of work to make things simple, and when you have a team that is unfazed by limitations, you get some really amazing and creative solutions, some of which I hope to see come out as open sourced projects here shortly."

Key in maximizing the value of the Obama campaign's IT spending was its use of open source tools and open architectures. Linux-particularly Ubuntu-was used as the server operating system of choice. "We were technology agnostic, and used the right technology for the right purpose," VanDenPlas said. "Someone counted nearly 10 distinct DBMS/NoSQL systems, and we wrote something like 200 apps in Python, Ruby, PHP, Java, and Node.js."

It also helped that the campaign, at least for internally developed applications, relied almost exclusively on Amazon Web Service for its infrastructure, eliminating a lot of the financial burden of infrastructure management. "For the applications built by the OFA [Obama for America] technology team, 99.999 percent were AWS hosted," VanDenPlas said, "purely because it was the best fit for what we were doing. As a whole, if you include privately hosted virtualized environments in the cloud architecture definition, I believe everything was 'cloud,' even down to our development environments running inside of Vagrant on our laptops."

The system configurations for the campaign's Elastic Compute Cloud (EC2) instances were created using the Puppet configuration management tool and were built as Debian packages kept in the campaign's own Advanced Packaging Tool (apt) repository-both for internally developed and third-party applications. As the number of applications and the scale of the campaign's AWS infrastructure use climbed, the DevOps team shifted to using Asgard-an open-source tool developed by Netflix to manage cloud deployments.

To help optimize applications, the OFA technology team used New Relic, a tool also used by the Romney campaign. "It is really a fantastic tool that increases your visibility into where your applications are spending time," VanDenPlas said. "They support the major languages we used (Python, Ruby, PHP) as well as the frameworks (Flask, Rails, Kohana)."

While AWS's tools were used for performance monitoring and to trigger automatic scaling-up of capacity, VanDenPlas said, much of the monitoring was handled by a suite of commercial and open source tools and home-grown code, "consisting of Cacti, Opsview, StatsD, Graphite, and Seyren, and a number of custom applications that continued to evolve right up until Election Day," VanDenPlas said.

To get better aggregated alerting and metric data, the team built a lightweight plugin for Nagios (the open-source basis of Opsview) in Python based on boto (the Python programming interface to AWS's services) and dotCloud's ZeroRPC messaging interface. "Using this," VanDenPlas explained, "we could constantly query thousands of nodes for near real-time statistics and feed them right back into the same alerting and monitoring system (Nagios) we used elsewhere."

Other performance monitoring and user experience data was collected using Chartbeat and Google Analytics. "Akamai also provided very useful statistics and logging," VanDenPlas said, "but these were mostly contextual rather than actionable." But, he added, the most heavily used monitoring system was "our community of internal and external supporters. The human factor in monitoring is huge. There are countless incidents where (OFA User Support Director) Brady Kriss notified us of pending problems derived from community help tickets."
The armor-plated cloud

The OFA engineering team also did a lot of work to ensure that they got the most out of Amazon's cloud architecture in terms of resiliency. As the election approached and the infrastructure demands surged, the engineering team took advantage of Amazon's multiple availability zones within its Virgina data center. "We built out a triply redundant, encrypted, and compressed WAN optimized tunnel between AWS regions," VanDenPlas said, "using a combination of OpenVPN, CloudOptimizer, and some DNS trickery."

The team shifted its domain name service to Amazon's Route 53 service, which uses latency-based routing to direct users to the host running in the AWS availability region with the shortest network trip time. That allowed the Obama team's application deployments to use "regionless" generic configuration settings, making deployments much simpler.

The centerpiece of the whole Obama campaign was its fundraising capabilities, without which all of the other applications may have been moot. The 2012 campaign's online donation system was a complete rebuild from the 2008 effort, VanDenPlas said, "a multi-region, geolocated, three facility processor capable of a per second transaction count sufficiently high enough that we failed to be able to reach it in load testing. It could also operate if every other dependent service had failed, including its own database and every vendor."

The Obama campaign's websites were also hosted on Amazon and hardened. The campaign's engineers built an application that created static HTML snapshots of the sites stored in Amazon's Simple Storage Service (S3); in the event of a Web server failure, requests would be instantly directed to the latest snapshot.

All of that redundancy was given an extra workout in the week before the election as Hurricane Sandy approached the East Coast. VanDenPlas said that a "complete hot replica of our entire infrastructure" was deployed to Amazon's primary West Coast data center in under 24 hours as a precaution.
Build, borrow, or buy

The tech team wasn't the only internal IT operation at Obama for America. The campaign ran its own data analysis shop and had its own army of Web designers and administrators. And with a payroll of over 1,000 people, the IT team had a lot of tech to support for an organization that had essentially a 24-month lifecycle.

That meant buying a lot of hardware and software. CDW, based outside of Chicago, was the go-to supplier for much of the campaign's computer equipment and boxed software purchases. Microsoft also sold $522,210 worth of software licenses to the campaign-which averages out to just under $500 per staffer.

With its investment in the cloud, the Obama campaign's Web hosting costs were much higher than Romney's. The largest cost, however, was content hosted by Blue State Digital, the social media and interactive advertising agency; the cost for hosting the internally developed applications in the Amazon cloud was a quarter of that:

And then there was the Obama campaign's outside technology help. As mentioned in our previous coverage of the Obama campaign, advertising company Blue State Digital and campaign software provider NGP VAN provided the largest chunks of Obama for America's technology consulting, and are most directly comparable to the over $14 million paid out by the Romney campaign to its digital firm, Targeted Victory. Even taken with the software and Web hosting expenses, the Obama campaign spent a seventh of what the Romney campaign spent on digital and an even smaller fraction of what Romney spent on voter and donor contact.

Return on investment

In the end, the deciding factor wasn't what the Obama campaign spent money on, but what it did with all that money. Insourcing gave the campaign a strategic flexibility that the Romney campaign lacked, as well as other intangibles that may have contributed to leading an efficient campaign. And the reduced reliance on outside consultants allowed the Obama campaign to direct capital toward places where it had a bigger impact-such as in advertising, where the Obama campaign outspent Romney by a factor of 5 to 1.

"This is the difference," VanDenPlas said, "between a well run professional machine and a gaggle of amateurs, posing in true Rumsfeldian fashion, who 'don't know what they don't know.' I would be shocked if such a chasm exists next cycle between the parties-these aren’t mistakes to be repeated if you want to do things like win elections."

Given the response from Republican partisans to the failure of Romney's campaign and to the apparent failure of its technology investments, the stakes for the next time-the mid-term elections in 2014, and the next presidential race in 2016-will be that much higher for Republican campaigns. It's doubtful they'll ignore the lessons learned this campaign season.


---Obama’s White Whale---
By Sasha Issenberg|Posted Wednesday, Feb. 15, 2012, at 11:28 AM ET
http://www.slate.com/articles/news_and_politics/victory_lab/2012/02/project_narwhal_how_a_top_secret_obama_campaign_program_could_change_the_2012_race_.html

How the campaign’s top-secret project Narwhal could change this race, and many to come.

On Jan. 22, a young woman in a socially conservative corner of southwestern Ohio received a blast email from Stephanie Cutter, a deputy campaign manager for Barack Obama. Years earlier, the young woman had registered for updates on Obama’s website, completing a form that asked for her email address and ZIP code. For a while, the emails she received from Obama and his Organizing for America apparatus were appeals to give money and sign petitions, and she responded to one that required that she provide her name. The emails kept on coming, rarely with anything an Obama supporter could disagree with, and certainly not the type of hard-edged political message that could scare one away.

But Cutter’s note was different. She boasted of a new administration rule that would require insurance plans to fully cover contraception as part of the president’s health care reform law, and encouraged her recipients to see the policy as reason to rally around Obama’s re-election. “Think about how different that is from what the candidates on the other side would do,” Cutter wrote. “Our opponents have been waging a war on women’s health-attempting to defund Planned Parenthood, overturn Roe v. Wade, and everything in between.”

It was a message that sat well with the young Ohioan who received it. She was single, liberal, sensitive to medical costs-but she had never told the campaign any of those things, and the one piece of information she had provided (her ZIP code) could easily mark her as the type of traditionalist Midwestern woman who would recoil at efforts to liberalize access to birth control. Indeed, she found it hard to believe that many other residents of her ZIP code would look as favorably upon a rallying cry to defend Planned Parenthood as she did.

Those who have worked with Obama’s data say that it is an email that would have never been sent in 2008. The campaign knew very little about the 13 million people who had registered for online updates, not even their age or gender or party registration. Without the ability to filter its recipients based on those criteria, the campaign stuck to safe topics for email blasts and reserved its sharp-edged messages for individual delivery by direct mail or phone call. In those channels, the campaign could be certain of the political identities of those it was reaching, because the recipients had been profiled based on hundreds of personal characteristics-enough to guarantee that each message was aimed at a receptive audience.

This year, however, as part of a project code-named Narwhal, Obama’s team is working to link once completely separate repositories of information so that every fact gathered about a voter is available to every arm of the campaign. Such information-sharing would allow the person who crafts a provocative email about contraception to send it only to women with whom canvassers have personally discussed reproductive views or whom data-mining targeters have pinpointed as likely to be friendly to Obama’s views on the issue.

From a technological perspective, the 2012 campaign will look to many voters much the same as 2008 did. There will not be a major innovation that seems to herald a new era in electioneering, like 1996’s debut of candidate Web pages or their use in fundraising four years later; like online organizing for campaign events in 2004 or the subsequent emergence of social media as a mass-communication tool in 2008. This year’s looming innovations in campaign mechanics will be imperceptible to the electorate, and the engineers at Obama’s Chicago headquarters racing to complete Narwhal in time for the fall election season may be at work at one of the most important. If successful, Narwhal would fuse the multiple identities of the engaged citizen-the online activist, the offline voter, the donor, the volunteer-into a single, unified political profile.

Traditionally, even the campaigns most intent on gathering varied types of data have had little strategy for getting all the information to work together. When computers started regularly appearing in campaign offices in the 1980s, different vendors developed distinct software packages for the varied work that went on there: volunteer-management programs, campaign finance and budgeting tools, voter-file interfaces that could spin off mailing labels or walk lists ready for neighborhood canvassers. The data were stored in different places, often through systems incapable of communicating with one another.

When Obama launched his candidacy in 2007, the departments of his campaign followed this pattern and developed their own repositories for the data they collected. State-level VoteBuilder databases could access rich information about people’s political activities that helped to refine statistical projections about their beliefs. The online databases developed by the firm Blue State Digital contained records of who registered for website and text-message updates, and how they responded to different appeals. The campaign’s fundraising team assembled its own list of donors. The field team had its database of volunteers, called Build the Hope.

“Every unit within the campaign had their little fiefdom and a chief. People were very proprietary about their data,” says a staffer at Obama’s 2008 headquarters. “They started as separate systems because that’s the way it works. No one ever thought System B would get useful data for System A-and we weren’t planning for the long run from the beginning.”

By the time campaign officials realized that they were agglomerating unprecedented volumes of political information-and that it would all become more valuable as it was allowed to mingle across categories-it was too late to rebuild their systems to make that sort of data-sharing easy. Even as the outside world marveled at their technical prowess, Obama campaign staffers were exasperated at what seemed like a basic system failure: They had records on 170 million potential voters, 13 million online supporters, 3 million campaign donors and at least as many volunteers-but no way of knowing who among them were the same people.



By Election Day in 2008, the campaign could come up with only what national field director Jon Carson described as a “Rube Goldberg” data apparatus, that depended on manually moving individual bits from one database to the next. Staffers would take the finance department’s contribution records and flag each donor’s record in a VoteBuilder database, but could rarely keep up with the volume of new people to track. New email signups came in even more quickly; at one point, as many as 100 volunteers enlisted into a virtual typing pool, copying Blue State Digital online contacts into the voter records.

Permanently linking the campaign’s various databases in real time has become one of the major projects for Obama’s team this year. Full data integration would allow the campaign to target its online communication as sharply as it does its offline voter contact. When it comes to sensitive subjects like contraception, the campaign could rely on its extensive predictive models of individual attitudes and preferences to find friendly recipients. In the case of Cutter’s blast, that might mean pulling email addresses only for those who had identified themselves as women on their registration forms and whose voter records included a flag marking them as likely pro-abortion rights.

More broadly, Narwhal would bring new efficiency across the campaign’s operations. No longer will canvassers be dispatched to knock on the doors of people who have already volunteered to support Obama. And if a donor has given the maximum $2,500 in permitted contributions, emails will stop hitting him up for money and start asking him to volunteer instead. Those familiar with Narwhal’s development say the completion of such a technical infrastructure would also be a gift to future Democratic candidates who have struggled to organize political data that has been often arbitrarily siloed depending on which software vendor had primacy at a given moment.

In a campaign that has grown obsessed with code-naming its initiatives, the integration project is known as Narwhal, after the tusked Arctic whale whose image (via a decal) adorns a wall adjacent to the campaign’s engineering department, as first reported by Newsweek. Narwhal remains a work-in-progress. Campaign spokesman Ben LaBolt refused to discuss the project, and the actual origins of their project’s code name are obscure, but at Obama’s headquarters the joke has become that reference to a mammal often called “the unicorn of the sea” has come to accurately describe an elusive quarry. Like much of what changes politics this year, Narwhal will remain below the surface, invisible to the outside world.

0 コメント: