Noel Grigsby caught two of the scoring passes and finished with 132 yards on eight receptions for San Jose State (9-2).
Riley Nelson threw for 335 yards and touchdown for the Cougars (6-5), but he was sacked four times. Cody Hoffman had 13 receptions for 155 yards.
Nelson hit David Foote for a 20-yard TD to cut the lead to six points with 2:36 remaining, and BYU recovered the subsequent onside kick. The Cougars drove to the Spartans' 21-yard line, but Nelson fumbled when hit by Keith Smith, and Vince Buhagiar recovered for San Jose State with 1:16 to play.
Twice earlier in the fourth quarter, BYU drove deep into SJSU territory but couldn't score.
Fales passed to Grigsby for scores of 51 and 18 yards. Chandler Jones caught the other scoring toss, an 8-yarder.
By winning the teams' first meeting in San Jose since 1968, the Spartans took a 10-6 lead in the all-time series.
BYU ends the season next Saturday at New Mexico State.