The average age at first marriage for a male is 23, so if Joseph made his first marriage in 1791, we would expect his birth to have been ca. 1768.

The average age at first marriage for a female is 19, so if she made her first marriage in 1791, we would expect her birth to have been ca. 1772.

Some secondary sources give James's birthplace as Isle La Motte, VT, while others give Champlain, Clinton Co., NY.  In the censuses, James gives his birthplace once as NY and twice as VT.  Given his father's location in the 1790 census, the VT location seems more likely.

Sources:

1.  Marriage Record:

2.  1790 Census Index/Images (online at Ancestry.com; Image #1 of 1; extracted by Diana Gale Matthiesen):¤•

1790 [actually 1791]

VT

Chittenden [now Grand Isle] Co.

Isle-Mott

Roll M637_12

p. 168

Ln. 7

Joseph Rasey

1-0-0-0-0

These data indicate:

Gender and Type	Age Class	Therefore Born	Individuals Inferred
1 free white male	16 or over	in or bef. 1774	= Joseph

There are only 14 households in Isle-Mott (total population = 47).

3.  P. William Filby, ed.  2005.  Passenger and Immigration Lists Index, 1500s-1900s.  Gale Research, Farmington Hills, MI (online at Ancestry.com):-

Name:	Joseph Rasey
Year:	1794
Place:	Quebec
Source Code:	1112.16
Primary Immigrant:	Rasey, Joseph
Annotation:	Date and port of arrival or date and place of declaration of intention. Extracted from records at the National Archives of Canada at Ottawa. Other genealogical information is also provided.
Source Bibliography:	CANADA, NATIONAL ARCHIVES OF. "Declarations of Aliens, Lower Canada, 1794-1811." In Across the Border (Carleton Place, Ontario, Canada). vol. 3:2 (Winter 1991), pp. 2-6.
Page:	6

Could this be our subject?  Could his emigration from the U.S. be the reason we do not find him again in the U.S. census?  We do know that at least one of his grandchildren returned to Quebec from NY and settled there permanently.  It would certainly be worth tracking down this complete record.

4.  Anon.  1891.  History of Franklin and Grand Isle Counties, Vermont.  D. Mason & Co., Syracuse, NY (online at HeritageQuest at Genealogy.com):

p. 656

In 1791 Isle La Motte was found to have sufficient population to warrant its organization and the election of its own town officers independ-

p. 657

ent of the proceedings of the proprietors under their charter rights.  On the 24th of March of that year the inhabitants met at the house of Nathaniel Wales, in pursuance of the customary warning, and elected officers for the year ensuing, as follows... tithingmen, Joseph Rasey...  During this same year was made the first census enumeration.  The count showed Isle La Motte to possess forty-seven souls; in 1800, nine years later, the population had increased to 135...

5.  LDS.  Family Search: Internet Genealogy Service:  AF - Ancestral File (online at FamilySearch.org).

6.  WorldConnect / Ancestry World Trees (online at RootsWeb.com/Ancestry.com).