If you are selling as an FBA seller, then all of your shipments will be handled by Amazon, if your shipments are lost or delayed then the responsibility will be on Amazon and the negative feedback regarding this will not affect your account.
If your shipments are being damaged during shipments then you need to update and upgrade your packaging so that the products do not damage during shipments.