In his London parish he regularly catechized the aged and the poor, and once a week distributed money among them, but changed the day in order to encourage constant attendance. In 1662, owing to his disagreement with the Act of Conformity, he was ejected from his living, and during 1662-71, with some other friends, raised a considerable sum of money to relieve the most needy of London's ejected clergy. In 1671 he read the life of Joseph Alleine, the nonconformist minister of Taunton whose purpose (interrupted by death) had been to evangelize Wales. Gouge determined to carry out his designs. He visited Wales, founded a number of schools, and became associated with Stephen Hughes (q.v.) in the distribution of Welsh books. To carry out his work, he appealed for subscriptions throughout England and Wales, and in 1674, a ‘trust’ was organized which was supported by the chief religious leaders of the time. The schools, which ultimately reached a total of about 300, were set up ‘to teach the poor Welsh children to read and write English, cast Accompts and repeat the Catechism.’ Many books were translated into Welsh for the use of the older people, including some of Gouge's own works, and in 1677, an octavo edition of the Welsh Bible, edited by Stephen Hughes, was published.

When Gouge d., 29 Oct. 1681, the schools came to an end, but the distribution of Welsh literature was continued for some time. The money left in the Welsh trust was used to found charity schools in London and the trust may be considered the pattern of the voluntary societies for the education of the poor in the 18th cent.