saved
Towards a Theory of Bugs: The Ruliology of the Unexpected
Hraness wrote this summary from a saved copy of the source. Quotations are taken word for word from the source.
gist
Stephen Wolfram argues that software bugs follow from computational irreducibility, not only from ordinary human mistakes. Even tiny Turing machines and cellular automata that seem to add one or double an input eventually fail on rare cases, because a program worth running must do something you cannot fully foresee. Tests and proofs cover only pockets of reducibility. Past those pockets, the only way to know the result is to run that case.
ideas
- A bug is a mismatch with a simpler specification. Relative to its own code a program cannot be wrong; it is a bug only against a shorter description, such as adding one, that a mind can hold.
- Irreducibility is the price of a program worth running. If every outcome were foreseeable you could write the answer down; once anything is unforeseeable, much else tends to be unforeseeable too.
- Proofs and tests only buy pockets of reducibility. Equational proofs and sampling can certify some properties, but a computationally irreducible process has no finite summary of everything it does, and rare failures can hide until enormous inputs.
- A well-fit language still cannot abolish surprise. Packing irreducibility into primitives, and seeing the computation as a diagram, helps; any program that must actually be run, including a trained net, can still do the unexpected.
quotes
“And in the end, we don’t get any fundamental advantage by trying to make a proof rather than just running the Turing machine.”
“Computational irreducibility is in a sense what makes computation powerful. But it’s also what makes it unpredictable.”
“Fundamentally the only way to make sure a program will always do what you want is to understand everything it can do.”