Inner panics could make it hard to trouble shoot the issues and for some
users it's not desirable.
The inner panics were left only when they are used to `assert!` during
development.
This reverts commit 9f91bc413fe20618bd7090829832bb074aab15c3 which
reverted the original patch which was merged without a proper review.
Fixes: #500.
f9758528f6