Turn the leaves over and look at the underside. If it has a light aluminum color, it's autumn olive. If it's as green as the top of the leaf, it's russian olive. Russian is not as invasive as autumn olive. Autumn is better for wildlife. Which is why it spreads so readily. By the way, they do make a nice tasting jelly.
15-20 foot tall is big for either of these olive species. They might not be doing good because of over-maturity.