The questions you ask are intriguing and of course any answer is guesswork right now. But I think it makes sense that there’s some kind of getting-to-know-you period, during which the device should learn its boundaries: what kind of behaviour is acceptable in this relationship (and that’ll probably be unique to each user), what the user responds to, etc. The idea of establishing norms via conditioning is itself very human so I think this could be congruous.

I suspect the best way to find out how these relationships will work is to build them and see what behaviour emerges. Easy enough, right? :)

“Mori’s hypothesis states that as a robot is made more humanlike in its appearance and motion, the emotional response from a human being to the robot will become increasingly positive and empathic, until a point is reached beyond which the response quickly becomes strongly repulsive.”

In Predictably Irrational Dan Ariely talks about economic and emotional reactions and how we flip between them. Essentially he argues that there are rules for proper conduct for each role but that at least some behaviour is mutually exclusive. At the moment I think I treat software as economic entities so I will accept paying, waiting for customer support, etc. I wonder how my attitude would change if my technology behaved like a person… Would I still tolerate errors? Would I find it intrusive? Maybe the software would have to go through a kind of courting dance – getting to know me and working out how friendly it should be. Would personality in my operating system endear it to me or trigger Mori’s repulsion?

Dunno. Thought provoking post!