Superintelligence

A rigorous analysis of what happens when machine intelligence surpasses human intelligence. Bostrom covers possible paths to superintelligence, the control problem, capability vs. motivation, and strategic considerations for humanity.

Dense and at times speculative, but the core argument — that aligning a superintelligent system's values with human values is extremely hard and extremely important — has proven influential. Essential context for anyone thinking seriously about AI safety.