A couple of options could be:
1. In 2001, we drove from Coventry left the car at Newport which is just over the bridge / border in Wales, then train'd it across to Cardiff Central (15mins) the stadium as others have said is very very close to the station, thus you avoid any traffic problems in the centre of Cardiff.
2. In 2002, I ran a supporters club coach from Rugby & Coventry to Cardiff, filled it easily will probably do the same this year.
As for direct train from Oxford to Cardiff, forget it you'll never get back. They do however operate a special shuttle after the meeting back to Newport, thus Option 1 could work for you if you were to be in a car.
My suggestion though, is to look out for a suporters club coach running from Oxford.