Depends on what you are looking to get out of it. If you want low grade, I've sawn some that were standing dead for 5+ years.
For your upper grades, you will want to get them before the borers set in. I've seen that go for several years. Might be different in your area. Walnut not as easily effected as the oaks. Sometimes the borers are patchy, where you can get some good wood.
You will lose the sapwood within the first year. If the bark doesn't pop off, then the sapwood will turn to dote. If it does, the sapwood will probably be stained.
We had a big problem with gypsy moth several years back. What a few guys tried to do was sell wormy oak. They could have found the market, but couldn't guarantee the quantity. We're talking trailer loads.
I've seen it used in wainscoat and paneling. Nothing wrong with it, and it looked good. It might be easier to market on a more localized level.