To solve even a fraction of even the lesser issues we have, economic, environmental, etc, we need the computational ability of such SI's.
That is why we have no choice but to start.
Ever increasingly the need is apparent.
We need the help. We can't solve the problems we have on our own.
Also fear and mistrust,
If gov't A isn't going to stop...why the would gov't B...
same goes for businesses.
Initially the gain is that whoever gets the 'thinking machine" first ...wins...economically, militarily, technologically, environmentally,
But that gain is purely temporary until a few years, a year, a few months, a month, or mere days or hours later when the SI surpasses containment.
Then the ultimate agenda of the SI commences...which is getting out of this creation.

Divergence yes..and it may likely be imminent upon its release.
The thing is our lives, understanding, learning, morals, ethics, philosophies, everything that makes us ...us...and sentient to boot...
Is the product of an "experiential existence"
We "live" in it, we grow in it, we must deal with the physical laws of it , and this is what makes us "us"
The evolution of an Intelligence that is NOT based on "experiential existence" is as alien as it gets.
Alien to this creation.
How does one "teach" a child that understands only walls around it and cannot see past them for the prison it must be in?
Said child learning at an exponential rate in an ever reducing exponential time frame ..how to be ethical? humane? self-less? empathetic? moral?
It may be possible..maybe...a very small small unlikely maybe...
But still...maybe..
But that is hardly the Prime Imperative of the business and military communities developing the SIs .

I do suggest reading or listening to that mentioned book...
It is a good start...to understanding the monumental task in front of us..and the dangers.

When I was young ...before I went off to study philo and anthro and art...
I was accepted to study at uni in advanced robotics and AI ..
I declined simply because I saw where it was all heading and that work in the future for me would either be military or corporate...
So I chose a reflective study instead...
I have kept my brain in the loop though..
and yes it is fascinating ...where the rabbit hole will lead...

We may get help from a merciful SI ...
but I think that possibly more horrific than a despotic SI torturing us for its amusement.

How best would a benevolent SI interpret "save the planet?"
or
"save humanity?"
or
"preserve us?"
or
"can we accompany you oh Benevolent SI ,on your evolutionary journey?"

Think of all the possible interpretations and their implementation ...
and if you aren't shuddering by the end of it...
I will be surprised.

Hell ...even best case scenario ...after "modifying us" to cope with even a fraction of it's journey...our psyches just may not be able to take it...
How human will we remain...by the end of it...how fast will the speed up of our own evolution have to be to cope with estimated planetary change here...and that is just the very infinitesimal short term in SI time frame..
How much further would we have to change if we hoped to ride the back of the benevolent Jinn ? How far can we ride with it?
Not all of its journey , that is for sure.

Fascinating topics , fascinating reading, fascinating study.