One Does Not Simply Forget What To Say