He married as his first wife, in 1661, Marie de Saint Simon (died 1670), daughter of Antoine de Saint Simon, marquis de Courtomer and Suzanne Madeleine, and by her, had three daughters:^[2]


He married, as his second wife, 12 March 1673, Suzanne de Beringhen (died 1731), daughter of Jean de Beringhen and Marie de Menoux, and by her had three sons and four daughters:^[2]


He died 19 April 1699 at Château de la Boulaye, near Évreux in Normandy. He was succeeded by his son Henri-Jacques Nompar de Caumont, duc de La Force, and then by his third son, Armand.^[2]


His widow was held prisoner for a spell at the convent of Évreux, and would end up in England as a staunch Protestant.^[3] Meanwhile the deposed King James II's Queen, Mary of Modena, was being part-way escorted by her distant cousin the Marquis de Beringhen to her final exile place in France, as a staunch Catholic. Suzanne refused to abjure (swear to be Catholic), and was confined to the convent until her husband's death, when the Earl of Jersey, English Ambassador at Paris, petitioned for her release and exile to England, where she was brought by his wife and given an apartment in St. James's Palace.


Her son Armand became an ardent Catholic whom local sources describe as bigoted. He was confirmed as beneficiary of all her lands in his mother's Will but he burned the library of Protestant books which his father had collected, and persecuted his tenantry with such great severity that some died.^[3]


Suzanne, the dowager duchess, was in late life among a group of gentrified Huguenots particularly from Bennes, including Isaac Gouicquet de St Eloy born in Plémy naturalised English in 1698, who emigrated with funds, to London. They breathed life into the volte-face French Protestant courtier-devoted Chapel Royal (i.e. Queen's Chapel part) at St James's and she was among those who generally lived in Sunbury on Thames.^[3] She died resident at St James's Place, Westminster. Sunbury, her gentile refuge, was an outer Middlesex village within a few hours ride of Westminster. St Eloy owned one of its three manors, Sunbury 1703-1718 and its neighbour in the parish which includes a French Street named after this period, Kempton Park, was somewhat earlier owned by Sir John Chardin and Esther Le Pigne, or Peigne, the uncle of Sir Philip Musgrave. In his family's hands that manor stayed for a longer period.^[3][4] She left by sworn direction, declared valid by a high court post mortem, in a society chest, a codicil legacy of £5,400, subject to intervening life interests, to London's French Protestant Hospital ('La Providence').^[5] Left among her 173 many minor legacies was to Mr. Le Blanc, minister of 'the little French church at Sunbury', perhaps a chapel at Kempton Park House or a side-concern of the main church, the sum of £12 p.a..^[3]