1.  1850 U.S. Census Every-Name-Index/Images (online at Ancestry.com, Image #67-68 of 542):  New York City (5th Ward), New York Co., NY, Roll M432_537, pp. 34A-34B, enumerated 5 Aug 1850, official enumeration date 1 Jun 1850 (extracted by Diana Gale Matthiesen):

334	492	Isaac Brown	69	M		Inspector of Lumber		Connecticut
		James   "	30	M		Artist		NYork
		Maria   "	22	F				"
[next page]
334	492	George W Brown	20	M		Silversmith		NYork
		Martha     "	60	F				Connecticut
		Abby Ganbrey	22	F				Ireland

Isaac BROWN is James's father.  Is Martha his mother or his aunt?

3.  1860 U.S. Census Every-Name-Index/Images (online at Ancestry.com, Image #47 of 95 — indexed "I. Sidney Brown"):  St. Louis P.O., City of St. Louis (Ward 6), St. Louis Co., MO, Roll M653_655, p. 313, PN 47, 109/169, enumerated 21 Jun 1860, official enumeration date 1 Jun 1860 (extracted by Diana Gale Matthiesen):

J Sidney Brown

41

M

Artist

NY

Living in some kind of boarding house or hotel.

4.  1870 U.S. Census Every-Name-Index/Images (online at Ancestry.com, Image #28-29 of 72):  New York City P.O., New York City (Ward 20, ED 15), New York Co., NY, pp. 454B-455A, PN 28-29, 83/267, enumerated 29 Jul 1870, official enumeration date 1 Jun 1870 (extracted by Diana Gale Matthiesen):

Chase Wm Henry	46	M	W	Hatter		$1500	Connecticut
"   Maria R	42	F	W	keeping House			New York City
"   Catharine A	12	F	W	going to School			Do
[next page]
Chase Cornelia M	11	F	W	going to School			New York City
"   Wm Francis	9	M	W	Cripple			Do
"   Julia	1	F	W	at home			Do
Brown Emma H	28	F	W	living with Aunt			Do
"   Amelia	24	F	W	Do			Do
"   James S	52	M	W	Artist		$1000	Do

Maria Rogers (BROWN) CHASE is James's sister.  Emma and Amelia BROWN are the orphaned daughter's of James's brother, Isaac BROWN, Jr.

5.  1880 U.S. Census Every-Name-Index/Images (online at Ancestry.com):  can't find.

6.  1890 U.S. Census:  the 1890 Census Population Schedules were destroyed.

7.  Trow's New York City Directory.  The Trow City Direcory Co., NY (online at Ancestry.com):

Year	Pg.	Name	Occupation	Loc1-work	Loc2-home
1867-8	134	Brown James S.	artist	765 B'way	-

8.  U.S. Passport Applications (online at Ancestry.com):  notarized on 1 May 1865 in NYC for James Sidney BROWN of NYC, b. 14 Apr 1819 in NYC.

9.  Anon.  3 Mar 1893.  "Artist Brown Very Ill." New York Sun (New York, NY) 60(184): 7.

ARTIST BROWN VERY ILL  —————  THE OLD DAGUERREOTYPER A PUBLIC CHARGE IN BELLEVUE.  —————  He Was Once Well Known in New York, and Was the Friend of Napoleon III., Samuel F.B. Morse, and Other Famous Men.       A white-haired man, whose back was bent with age and whose clothes had seen much wear, clambered stiffly down the narrow stairs of the brick building at 50 Fulton street at 6 o’clock on the morning of Washington’s birthday and tottered along in the storm that made out of doors dismal at that hour.  He slipped on the slushy crossing, and before he had gone very far fell on his face.  He was found by a policeman lying nearly insensible, with blood flowing from two bruises above his temple.  An ambulance took him to Bellevue Hospital, where the old man, dazed by his severe fall, could give no account of himself.  He was considered insane by the doctors, and placed in ward 26 as a patient suffering from dementia.  The doctors could find nothing in his possession that gave any clue to his identity, and the patient refused to talk about himself when he had recovered his speech.       Alonzo J. Drummond, a photo-engraver living at 1601 Avenue A, went to Police Head-quarters on Feb. 23 and begged the police to try and find the old man.  Drummond said that he was James Sidney Brown, an artist, and had lived for four years in an apartment in the third loft of 50 Fulton street.  The police tried for three days to find out what had become of the aged artist, but failed, as there was no record at Bellevue to show his identity. His whereabouts were discovered on Feb. 27 through a letter from the supposed insane patient, written in a perfectly rational style.  In it the old artist said that he was lying in the hospital at death’s door, and begged Mr. Drummond as an old friend to see that he had a decent burial in “the Varick street graveyard,” where, he wrote, some of his family slept.  The letter said that in return for this kindness Mr. Drummond might have the contents of two leather trunks, fully half a century old, that were in the third loft of 50 Fulton street.  The writer said that they would be found to be valuable.       Mr. Drummond hurried to Bellevue and found his old pensioner lying very ill, in charge of Dr. Douglass.  Tears came into the old man’s eyes, and he seized Drummond’s hand feebly.       “It’s the same old Jimmie Lenney,” he said, “will you see that I get a Christian burial when I go?”      Mr. Drummond resolved, if possible, to secure the removal of the patient to a home where his surroundings might be more comfortable.       Yesterday morning he opened the two trunks in the Fulton street loft, and found a multitudinous array of oil paintings, water colors, and wood engravings, pencil sketches, boxes of paints, books on art, and hundreds of letters.  There were scores and scores of daguerreotypes in excellent preservation.  Artist Brown would never permit any one to see these and would lock himself in the loft whenever he had occasion to open the trunks. With the aid of the letters and other memoranda, Mr. Drummond was able to trace out the story of his old pensioner’s career.       “Mr. Brown,” he said, “was born at the corner of Morris and Greenwich streets on April 15, 1820.  His father was a Greenwich street fruit merchant of some means.  He apprenticed young James to a silversmith at Fulton and Dutch street, but the boy was fonder of making sketches than fussing over silverware.  His catechism issued in 1829 by the Sunday school attached to old Trinity shows this.  His brother, Charles Brown, who was a brother-in-law of “Billy” Moloney, the Reading Clerk of the Board of Aldermen, was able, through Clerk Moloney’s influence, to get young Brown admitted to the antique school of the National Academy of Design, and he became a student in 1841.  He was a proficient student, and, after graduation he set up a portrait studio at 181 Broadway.  He was skilled at painting in water colors, and was one of the founders of the American Society of Water Color Artists.       “When the art of daguerreotype making was introduced it interfered seriously with portrait painting, and Mr. Brown accepted an engagement from Mathew Brady, a noted photographer, whose studio in Broadway, below St. Paul’s Church, is pictured in old woodcuts of antebellum days.  He had, previous to this, declined an invitation from Commodore Perry to accompany the Japan treaty exhibition.  Perry wanted him to make photographs of this historic trip.  Mr. Brown’s knowledge of chemistry and art enabled him to turn out exceptionally fine daguerreotypes.  He excelled in the art of posing his subject.  Some specimens of his work were sent to the London Exposition of 1851, and won the gold medal over competitors from other countries.       “He went back to his painting of portraits and landscapes after winning this medal and founded the Ruskin life class of the National Academy, where the students worked exclusively with the brush instead of the pencil.  He went to St. Louis later, still as a photographer and portrait painter, and his letters show that he made the acquaintance of Gen. Beauregard and other distinguished Southerners, and painted portraits and took daguerreotypes of them.  The outbreak of the war compelled him to sell out his photographic business at a sacrifice, and with the money he received he went to Paris and studied with Thomas Conture, the French painter, who highly commended his ability as an artist in water color.       “He returned to New York after the war, but found business rather dull, and was obliged to live frugally.  His last employment was as assistant to R. A. Lewis, a photographer in old Chatham street.  Lewis’s death left him practically penniless.  He had been a wood engraver for the Bible Society and for the Harpers and Frank Leslie, but he always lived up to his means.  He belonged to the Seventh Regiment for four years.  I have discovered from his correspondence that he was acquainted with the Emperor Napoleon III., Samuel F. B. Morse, Charlotte Cushman, Edwin Forrest, Gen. McClellan, and a number of other Northern Generals.”       Mr. Drummond spent an hour yesterday afternoon showing some of the treasures in the battered trunks that the old artist guarded so zealously.   There were many daguerreotypes of Mr. Brown, showing him as an intellectually handsome man, with flowing black hair and a full beard.  His forehead was high, and the expression of his face kindly.  There were studies in oil made in the Tuileries and the Louvre, and many water colors of children at play, resembling the work of J.G. Brown.  These things show that the old artist was particularly fond of gamin life.       Among the first things found in the trunks were copies of the famous wood engraving of Actor F.S. Chanfrau as Mose, for which Chanfrau posed to Mr. Brown.  There were wood engravings of scenes from other local plays of old New York life, and sketch books, in the pages of which the artist had jotted down characters whom he saw in walks in the Battery, along Broadway, and in City Hall Park.  They included all sorts of street gamins and peddlers, and the sketches were well done.       One of the curios was a tin box of Windsor & Newton colors made to order, with the colors in triangular pockets instead of square spaces.  The triangular pockets permitted a larger stroke of the brush and save time in getting the color in the camel’s hair.  There were numerous charts showing that Mr. Brown had been at work on a system of charts intended to teach beginners the principles of color blending.  The charts were ingenious, and made up of sunshine and shadow hues, and the effects of running water, still water, and the like.  Other charts showed that he had been experimenting to improve the method of taking daguerreotypes.       “These things,” said Mr. Drummond, “ought to bring in enough money when sold to keep Mr. Brown in comfort during the short time that he will live.  I have eight children and a wife to support or I would undertake to keep him unaided.  He needs medical attendance, and a man with such a career certainly deserves to spend his last days somewhere besides on a cot in Bellevue.”

Lith. & Published by E. & J. Brown 140 Fulton St. N.Y.

The "Varick Street graveyard" (a.k.a., St. John's Chapel cemetery, Trinity Parish) no longer exists; the property is now a public park, specifically, a baseball field.

10.  Anon.  Sunday, 5 Mar 1893.  "Will Never Be Himself Again."  The World (New York, NY), p. 24:

11.  Anon.  Sunday, 12 Mar 1893.  "Artist Brown's Pictures."  The World (New York, NY), p. 24.
There's no evidence that he was born on "Varick Street."

12.  Chas. F. Chandler & Arthur H. Elliott, eds.  1893.  "Obituary." Anthony's Photographic Bulletin (New York City, NY) 24(1): 282 (online at books.google.com):

p. 282

OBITUARY      We are under the painful necessity of recording the demise of James Sidney Brown, an old-time painter, engraver and daguerrerotyper...      Mr. Brown was born April 14, 1819.  His father was a Greenwich street fruit merchant of some means, and apprenticed young James to a silversmith; but the boy was fonder of making sketches than of working over silverware.  His brother finally got him admitted to the antique school of the National Academy of Design in 1841.  He was a proficient student, and after graduation set up a portrait studio at 181 Broadway.  He was one of the founders of the American Society of Water-Color Artists.  When the art of daguerreotyping was introduced Mr. Brown took it up, and was very successful in it.  He subsequently went back to painting again, however, and went to St. Louis, from which place he returned to New York after the War, but found business rather dull.  His last position was with R.A. Lewis, a photographer in Chatham street.  Lewis' death left him practically penniless.  He had been a wood engraver for the Bible Society and Harpers for some years.

13.  Brown, James Sidney (online at craigcamera.com/dag/bo_table.htm; extracts by DGM):

(1819-1893), daguerreian, artist and engraver.  States he was "probably" born in Newburyport.  Apprenticed as a silversmith.  Was M.B. Brady's principal operator by 1843.  In 1846, in partnership with brother, E.S. Brown.

14.  Peter E. Palmquist & Thomas R. Kailbourn.  2005.  Pioneer Photographers from the Mississippi to the Continental Divide:  A Biographical Dictionary, 1839-1865.  Stanford University Press, Stanford, CA.  Pp. 130-131.

15.  LDS.  Family Search:  Family Trees (online at familysearch.org):  not found, as of 13 Aug 2015.

16.  Public Member Trees (online at Ancestry.com):  not found, as of 13 Aug 2015.

17.  WorldConnect / Ancestry World Trees (online at RootsWeb.com/Ancestry.com):  not found, as of 13 Aug 2015.