I used to ask a similar question about capitalism. Why does there have to be a perpetual growth? Why is "not worse than last year" never good enough?

I eventually cottoned on to the answer. It is quite simple, really. If you don't grow, someone else will. And they will outgrow and eventually smother and destroy you. So you are forced to grow just to keep up. The same reason why trees are tall.

It may be a short-term view but that is how the nature works too. So many evolutionary dead ends because in any given population, only those individuals who have an immediate advantage win, even if that immediate advantage may be their long-term undoing.

I thank you a thousand times for that! If only I had a penny for every time some religious nut tried lecturing me how we would all descent into a lawless chaos without religion. In reality, we need all that you mention above just to survive. Species where the majority of individuals kill each other on the spot do not last very long. Of course there are a few individuals who deviate from the norm, but that is exactly the point - they deviate, they are not the norm.

I am glad you added that last bit. Alien to us, indeed. Not because it does not have "experiential existence". It does, only not the same as ours.

Polar bears, duck-billed platypuses, red sequoyas and smallpox viruses have their own "experiential existence" too, but because we do not share it with them, we do not give two squats about them. The same fate may befall us at the hands of SI. It may not destroy us out of hate or anything personificated like that. It may be that we are simply like insects to them. We do not kill the moles and earth worms on purpose when we build a new house. We kill them as a collateral damage, not giving them much thought.

The kind of SI discussed above is waaaay beyond self-driving cars. You can compare the complexity of SI to that of a self-driving car like comparing the complexity of New York with all its transport and communication links, businesses, inhabitants and their pets and their pets' dreams... to that of a straw hat. Sure, a straw hat is complex enough, but does it really compare?