Yes, he did miss out some years altogether - 1957 and 1964 - 68 - but for the other 17 years he only rode for Wimbledon (apart from the two matches for Coventry in 1972), which must be the record.
There was, of course, Tommy Price. Wembley 1935 - 1956, a total of 22 years, but, as with Bert Spencer, there were 7 years lost because of the War.