Given the age-gap between William and Eliza, we would immediately suspect there was an earlier wife.  Given the older children, plus the gap between Norman and William (Jr.), plus the lack of children under five years of age in the 1830 census, it's hard to avoid the conclusion that William had to have had an earlier wife.

— born in Chautauqua or Cattauraugus Co., NY:
.  (Daughter D) BROWN, b. 1830-35
.  Jane BROWN, b. 1834/5
.  Charles BROWN, b. 1835/6
.  George BROWN, b. 1838/9

— born in Cattauraugus Co., NY
.  Hiram L. BROWN, b. 1841/2
.  James F. BROWN, b. 1842
.  Frances Melissa BROWN, b. 1847/8

1.  Marriage Record:

2a.  1820 Census Index and Digital Images (online at Ancestry.com; Image #3 of 11; extracted by Diana Gale Matthiesen):

1820

NY

Chautauqua Co.

Chautauqua Town

p. 42

Ln. 14

William Brown

000010-00100-0-100

These data indicate:

No. & Sex	Age Class	Therefore Born	Individuals Inferred
1 male	26-44	1775-1794	= William (b. 1790)
1 female	16-25	1794-1804	= Wife-1 (b. ca. 1794) or Eliza (b. 1801/2)
1	person engaged in agriculture

2b.  1820 Census Index and Digital Images (online at Ancestry.com; Image #5 of 6; extracted by Diana Gale Matthiesen):

1820

NY

Chautauqua Co.

Portland

p. 133

Ln. 19

Munson Brown

100110-10101-0-001

These data indicate:

No. & Sex	Age Class	Therefore Born	Individuals Inferred
1 male	9 or under	1810-1820	= Son A
1 male	19-25*	1794-1801	= brother? brother-in-law?
1 male	26-44	1775-1794	= William? Munson (b. 1790)
1 female	9 or under	1810-1820	= Daughter A (b. 1815-20)
1 female	16-25	1794-1804	= Wife-1 (b. ca. 1794) or Eliza (b. 1801/2)
1 female	45 or over	in or bef. 1775	= mother? mother-in-law?
1	person engaged in manufacture
*In the 1820 Census, the third column is age class 16-18 and the fourth column is age class 16-25; therefore, any individual in column three is duplicated in column four.  By subtracting the number in column three from the number in column four, you can create an age class "19-25."

Is Munson our subject's middle name?  The male in his teens/20s is too old to be William's son.  Listed five lines from Samuel MUNSON.

2.  1830 Census Index and Digital Images (online at Ancestry.com; Image #27 of 30 — indexed as being in "Westfield," but there is nothing on the pages, themselves, that I can see indicating anything other than Chatauqua County; extracted by Diana Gale Matthiesen):

1830

NY

Chautauqua Co.

Westfield?

p. 486

Ln. 15

William M Brown

010 020 - 021 010

These data indicate:

No. & Sex	Age Class	Therefore Born	Individuals Inferred
1 male	5-9	1820-1825	= Norman (b. 1821/2)
2 males	20-29	1800-1810	= ? = ?
0 males	30-39	1790-1800	= William (b. 1790)
2 females	5-9	1820-1825	= Daughter C = Daughter B
1 female	10-14	1815-1820	= Daughter A
1 female	20-29	1800-1810	= Eliza (b. 1801/2)

William is in the wrong age class, yet it's hard to doubt this record is our subject, especially if the birthplace of William Jr. is as asserted (Portland is near Westfield).  The fact that there are no children under five in the household suggests William and Eliza are newlyweds and that the children shown are by a former wife.

3.  1840 Census Index and Digital Images (online at Ancestry.com; Image #1 of 7; extracted by Diana Gale Matthiesen):

1840

NY

Cattaraugus Co.

Cold Spring Twp.

pp. 91B-92A

William M Brown

110 100 100 - 020 101 000

0102000

These data indicate:

No. & Sex	Age Class	Therefore Born	Individuals Inferred
1 male	4 or under	1835-1840	= George (b. 1838/9)
1 male	5-9	1830-1835	= Charles (b. 1835)
1 male	15-19	1820-1825	= Norman (b. 1821/2)
1 male	40-49	1790-1800	= William (b. 1790)
2 females	5-9	1830-1835	= Jane (b. 1834/5) = Daughter D
1 female	15-19	1820-1825	= Daughter C
1 female	30-39	1800-1810	= Eliza (b. 1801/2)
1	person engaged in agriculture
2	persons engaged in manufactures and trades

Listed 6 lines from Ephraim BROWN.

4a.  1850 Census Index/Images (online at Ancestry.com; Image #1 of 14):  South Valley, Cattaraugus Co., NY, p. 9A, 5/5, enumerated 18 Jul 1850 (extracted by Diana Gale Matthiesen):

William M. Brown	60	M		Farmer		Conn
Eliza Brown	48	F				Canada
Charles Brown	14	M				New York
George Brown	11	M				"
Hiram Brown	8	M				"
James Brown	8	M				"
Melissa Brown	2	F				"
Jane Brown	15	F				"

I'm assuming Jane is out of order by accident.  Living two doors from William's son, Norman BROWN. 

4b.  1850 Census Index/Images (online at Ancestry.com; Image #5 of 15):  Cold Spring, Cattaraugus Co., NY, pp. 3A-3B, 33/37, enumerated 16 Jul 1850 (extracted by Diana Gale Matthiesen).  For the full record, see David WRIGHT's family group sheet. 

David Wright	31	M		Farmer		New York
[David's wife and children]
[next page]
William Brown	18	M		Lumberman		New York

5a.  1860 Census Images (online at Ancestry.com; Image #1-2 of 20):  Randolph & East Randolph P.O., Cold Spring, Cattaraugus Co., NY, pp. 657-658, PN 1-2, 138/137, enumerated 25 Jun 1860 (extracted by Diana Gale Matthiesen):

William M Brown	28	M		Lumberman	1200	2500	New York
Emeline Brown	27	F		Housewife			Do
Frank Brown	3	M					Do
Grace Brown	1	F					Do
[next page]
Hamilton Robb	23	M		Mill Laborer			Penn
Charles Cooper	19	M		Do			New York
John Mattoon	20	M		Do			Do
William Brown	69	M		Farmer			Connecticutt
Thaddeus S Van Scoter	16	M		Mill Laborer			New York

Living next door to William's wife, Eliza BROWN (see next record).

5b.  1860 Census Images (online at Ancestry.com; Image #1 of 20):  Randolph & East Randolph P.O., Cold Spring, Cattaraugus Co., NY, p. 657, PN 1, 137/136, enumerated 25 Jun 1860 (extracted by Diana Gale Matthiesen):

Eliza Brown	55	M		Housewife			Canada
Hiram L Brown	18	M		Mill Laborer			New York
James F Brown	17	M		Do			Do
Frances M Brown	12	F					Do

Living next door to Eliza's husband, William M. BROWN, Sr., who is living with their son, William M. BROWN, Jr.

6.  1870 Census Images (online at Ancestry.com):  William is deceased.  Eliza not found, and I checked every line of Cold Spring and Randolph, though I did find son, William M. BROWN, Jr., in Randolph.

7.  1880 United States Census and National Index (LDS Family History Resource File: CD-ROM Library):

Census Place: Cold Spring, Cattaraugus, New York
Source: FHL Film 1254811; NARA Film T9-0811; Pg 88A
	Relation	Sex	Marr	Race	Age	Birthplace	Occupation	Fa	Mo
William FAY	Self	M	M	W	38	NY	Farms	NY	NY
[William's wife and children]
Eliza BROWN	Other	F	W	W	75	CANADA	House Keeper	NH	NH

8.  William Adams, ed. 1893. Historical Gazeteer & Biographical Memorial of Cattaraugus Co., with Map and Illustrations.  Lyman, Horton, & Co., Syracuse, NY.  1164 pp.  (facsimile reprint in 1995 by Higginson Books, Salem, MA).  On p. 1042:

Town of Randolph ABSTRACT of Biographical Sketch: 1. William M. BROWN, Sr. - b. Connecticut; settler of both South Valley and Cold Spring; d. 1863, Cold Spring     +?     1. William M. BROWN, Jr. - b. 1830, Portland, Chautauqua Co., NY; 1870, Sheriff of Cattaraugus Co.; Supervisor of Cold Spring for 6 years         +Emeline M. WOODWORTH, d/o Madison WOODWORTH of Cold Spring; m. 1855; had four children     2. Norman BROWN