I always used eat the whole apple but the stem. Don't know if it caused any harm or not. We will never know.
As for almonds I happen to have an almond tree in my yard. It actually looks like a peach tree, has blossoms like a peach tree and bears fruit that look like small green peaches. They do not fill out like regular peaches and do not get any color. The pit looks exactly like a peach pit and inside is the almond which looks exactly the seeds inside peach pits. They are a favorite for the red squirrels though and I have to keep the population in check in order to get any for myself.