Aside from your own and your opponents drivers, races will feature 20 other cars. The difficulty of these cars is determined by which Series you race in. 

As you gradually progress through each Series, the competitiveness of the other drivers will also gradually increase. 

To stay competitive with the other drivers and your opponent you'll need to keep unlocking higher series drivers and components, while investing in them to power them up.