Decentralized Learning in Stochastic and Mean-Field Games