Whatsapp is not open source. It cannot be trusted that there are no backdoors, such as phoning home with the encryption key, and it can't be trusted that the cryptographic protocol has been implemented correctly. Whatapp in fact has a history of cryptographic blunders.
There is no way to verify the authenticity of the other device. A silent MITM attack can (or may already) be easily executed, where you actually encrypt your message to an imposer that then passes it to the other party.
If keys are stored on the device and Sync / iCloud is enabled, the app data will be backed up to Google's servers or Apple's servers, making it no longer end to end encryption.
There are tons of better solutions if you're interested in keeping your messages private, including (shameless plug) (https://subrosa.io).