Apple 2.0

Covering the business that Steve Jobs built

In Piper Jaffray re-test, Siri raises her grade from a D to a C

December 20, 2012: 9:05 AM ET

In a quiet room, the latest version understood 91% of questions, correctly answered 77%

377704456396FORTUNE -- Siri is a little like the weather. Everybody complains about it, but only Piper Jaffray's Gene Munster seems to do anything.

In June he put the iOS 5 version of Apple's (AAPL) voice-activated personal assistant to the test -- asking her 1,600 questions, 800 in the streets of Minneapolis and 800 in a quiet room.

The results left much to be desired. In the best of circumstances (i.e. no street noise), Siri correctly interpreted 89% of the questions and correctly answered 68%. (Typical exchange, as Siri heard it: When is the next Haley's comment?  "You have no meetings matching Haley's".)

On Thursday, Munster reported the results of a quiet-room (no street noise) re-test of the "improved" iOS 6 version. This time Siri understood 91% of the questions and correctly answered 77%.

In school-grade terms, she's gone from a D to a C.

(For reasons Munster doesn't explain, he's raised Siri's June score from 68% to 76%, making the improvement look more modest than it was. Perhaps he's now grading on a curve.)

Click to enlarge.

Click to enlarge.

Munster also reported on the performance of Google Now, Google's (GOOG) answer to Siri. Overall, Siri did slightly better, as the bar chart at right shows. But each program has its strengths and weaknesses.

"Siri's biggest strengths," Munster writes, "are in local discovery and OS commands. Siri's biggest weaknesses are in commerce and information. For Google, unsurprisingly, navigation and information are its strongest points, while commerce was weaker. Google Now also was significantly weaker in OS commands. For example, Siri enables full control of the music application via Siri, but Google Now does not understand all song change/pause commands. We believe Google will eventually strengthen its ability to control the OS in future launches. Commerce was an area in which we viewed both solutions as weak."

Below: Munster's side by side comparison.

Screen Shot 2012-12-20 at 8.33.48 AM

Join the Conversation
About This Author
Philip Elmer-Dewitt
Philip Elmer-DeWitt
Editor, Apple 2.0, Fortune

Philip Elmer-DeWitt has been following Apple since 1982, first for Time Magazine, and now on the Web for Fortune.com.

Email | @philiped | RSS
Current Issue
  • Give the gift of Fortune
  • Get the Fortune app
  • Subscribe
Powered by WordPress.com VIP.