The three young deceased children are hypothetical, but likely.

1.  Marriage Record:

2.  1851 Canadian Census Every-Name-Index/Images (online at Ancestry.com, Image #82 of 197; indexed "Roseberger"):  Williamsburg, Dundas Co., Canada West (Ontario), p. 17, official enumeration date 12 Jan 1852 (extracted by Diana Gale Matthiesen):

1851:  for an explanation of the column headings, please see What the Numbers in the Canadian Census Mean (only columns 1-9 extracted.).
Row	1 Name	2 Occupation	3 Birthplace	4 Religion	5 Residence if out of limits	6 Age next birthday	7 Male	8 Female	9 Marital Status
27	Martin Roseberger	Farmer	U States	Lutheran		45	/		M
28	John Rosenberger	Labourer	Canada	Do		14	/		S
29	George Rosenberger	Do	Do	Do		13	/		S
30	James Rosenberger	Do	Do	Do		10	/		S
31	Alex. Rosenberger	Do	Do	Do		6	/		S
32	Gilbert Rosenberger	"	Do	Do		4	/		S
33	Betsy Rosenberger	"	Do	Do		47		/	M
34	Mary Rosenberger	"	Do	Do		24		/	S
35	Kathrine Rosenberger	"	Do	Do		20		/	S
36	Caroline Rosenberger	"	Do	Do		19		/	S

3.  1861 Canadian Census Every-Name-Index/Images (online at Ancestry.com, Image #279 of 430):  Township of Williamsburgh, Dundas Co., Canada West [Ontario], p. 67, official enumeration date 14 Jan 1861 (extracted by Diana Gale Matthiesen):

1861:  for an explanation of the column headings, please see What the Numbers in the Canadian Census Mean (only columns 1-10 extracted).
Row	1 Name	2 Occupation	3 Birthplace	4 Married during year	5 Religion	6 Residence if out of Limits	7 Age next birthday	8 Male	9 Female	10 Marital Status
24	Martin Rosenbarg	Coper	United St		luthrin		54	/		m
25	Ellisabath  do		u C		"		54		/	m
26	Cathrin     do		"		"		26		/	S
27	Geramah?    do		"		"		1	/
28	John        do	lab	"		"		23	/		S
29	George      do	Carpinter	"		"		21	/		S
30	James       do	lab	"		"		18	/		S
31	Elexandar   do	lab	"		"		14	/		S

Elizabeth is too old to have birthed Jeremiah.  Is Cathrine an unwed mother?  The fact that Jeremiah was listed after her, and not at the end of the listing, suggests that is the case.

4.  1871 Canadian Census Every-Name-Index/Images (online at Ancestry.com, Image #50 of 124):  Township of Williamsburgh, Dundas Co., Province of Ontario, p. 2, official enumeration date 2 Apr 1871 (extracted by Diana Gale Matthiesen):

1871:  for an explanation of the column headings, please see What the Numbers in the Canadian Census Mean (missing columns contained no data).

5

6

7

8

9

10

11

12

13

14

15

17

18

19

5

6

Rosenbarge Martin

M

64

O

Luthrin

Dutch

Farmer

M

"     Elisabith

F

60

"

"

"

M

"     James

M

25

"

"

"

Farmer

/

/

"     Cathrin

F

35

"

"

"

/

"     Geremay

M

10

"

"

"

/

"     Edman

M

10/12

June

"

"

"

Elizabeth is too old to have birthed either Jeremiah or Edmond.  I'm guessing they're grandchildren.  Martin was born in the USA, not Ontario.  Such mistakes are not uncommon; the informant wasn't necessarily Martin.  The fact that Catherine is listed after the younger James points to the children being hers.  Otherwise, why not list her before James?

5.  1881 Canadian Census Every-Name-Index/Images (online at Ancestry.com, Image #76 of 98):  Williamsburg, Dundas Co., ON, p. 6, official enumeration date 4 Apr 1881 (extracted by Diana Gale Matthiesen):

1881:  for an explanation of the column headings, please see What the Numbers in the Canadian Census Mean (missing columns contained no data).
5	6	7	8	9	11	12	13	14	15
25	25	Rosenburger Martin	M	75	O	Lutheran	German	Farmer	M
		"      Elizabeth	F	72	"	"	"		M
		"      Cathrine	F	39	"	"	"
		"      James	M	37	"	"	"	"
		"      Jeremiah	M	20	"	"	"	Farmer
		"      Edmond	M	10	"	"	"	"

Again, Elizabeth is too old to have birthed Jeremiah or Edmond; and again, Martin was born in USA, not Ontario. 

6.  1891 Canadian Census Every-Name-Index/Images (online at Ancestry.com):  a widowed Marten ROSENBERG (æ 84, b. USA, parents b. USA) is living in Williamsburg, Dundas Co., ON, with his children and grandchildren.

7.  1901 Canadian Census Every-Name-Index/Images (online at Ancestry.com):  can't find.

8.  Ontario Deaths, 1869-1937 (online at FamilySearch.org):

• death of Elizabeth ROSENBERGER (æ 80, b. ?) on 21 Jun 1885, Dundas Co., ON • death of Widow Martin ROSENBERG (æ 87, b. USA) on 12 June 1891 in Williamsburg Twp., Dundas Co., ON; • death of Widow Caroline HILL, d/o Martin ROZENBURG (æ 85y 3m 5d, b. Williamsburg Twp.), d. 24 Aug 1919, Winchester, Williamsburg Twp., Dundas Co., ON; bur. Maple Ridge.