People have been referring to white plant excretions as “milks” for hundreds of years. Coconut milk, milk of magnesia, etc. Hell, almond milk was a popular ingredient in the Middle Ages.

But you’ll also notice that language has changed a lot since the Middle Ages. This is a natural process, and no amount of prescriptivism or pedantry will stop it. Calling things that are not dairy — but that we use in all of the exact same ways as dairy milk — “milks” is not a problem.