More or less what you said seems right to me. I surf too, so the waves that each sport "likes" are definitely different. I think most people mainly look at the wind first with the waves being secondary. If you have 15+kts of wind in a good direction, you know it will be a pretty decent downwinder no matter what any groundswell might be doing. Groundswell might affect the currents and make for some chop, but that's something that depends on where you are and the tides too.
In SoCal you have arguably the most wind and wave forecasts in the world, so its just a matter of combining the experience between the readings from various buoys, stations, apps/websites and correlating that to what you will see in the real world. You will find some over estimate, some under, some skew, etc. so you just need to note that. A physical journal/notebook is actually really helpful.
NOAA buoys 46622 and 46624 are good for real-time data.