From Wikipedia: In 1641, on the eve of the English Civil War, Alathea and her husband, their son, Viscount Stafford, and his wife fled to the Netherlands. She commissioned an inventory of the contents of Tart Hall, her home on the margins of St James's, which included a chamber known as the Dutch Pranketing Room.
Alathea and her husband had been appointed to escort Marie de' Medici, Dowager Queen of France, who had been in England to visit her daughter Henrietta Maria, Queen of England (and also because she was on the outs with her son, the King of France) to Cologne and safety.
Lady Arundel was not prepared to wait for Marie de' Medici and with characteristic decisiveness set off for the Continent on her own, the reason being, so it was said, that she had a 'mania' for travel. Alethea went straight to Utrecht and met there with her husband. When he accompanied Marie de' Medici to Cologne, Alethea tried to persuade Urban VIII to allow her to enter a Carthusian monastery. In 1642 her husband accompanied the Queen and Princess Mary for her marriage to William II of Orange and left straight for Padua.
She lived in Antwerp, but moved to Alkmaar, after her husband died. She invited Franciscus Junius, for thirty years in their service, to rearrange the collection of books. Then she moved to Amersfoort (1649), and rented a pied-a-terre in Amsterdam at Singel 292, an elegant house, with a courtyard facing Herengracht.
When the Earl of Arundel died, Alethea inherited the collection of 600 paintings and drawings including works by Dürer, Holbein, Brueghel, Lucas van Leyden, Rembrandt, Rubens, Van Dyck, Raffaello da Urbino and Titian. There were 181 works with no attribution; 200 statues and 5,000 drawings, which he had bought with her money. His debts (or the collection) were estimated £100.000. She inherited Arundel Castle and Arundel House. Her eldest son argued three years in court against his father's will.
On 3 June 1654 Alethea died in Amsterdam without leaving a will and a compiled and far from clear inventory was made. The inventory consisted of 36 paintings by Titian, 16 by Giorgione, 19 by Tintoretto, 11 by Correggio, 17 by Veronese 12 by Rafaello and five by Da Vinci.
«The Fate of Arundel’s Pictures
The Arundels made arrangements for their pictures to go to the Low Countries where they arrived about 1643. Examples of the art to arrive in the Low Countries included Holbein’s Dr Chambers (Vienna) and studio versions of Titian’s Three Ages of Man. The impact of these two artists in Antwerp must have been considerable where connoisseurship was enthusiastically pursued. Towards the middle of 1645 Arundel left Antwerp for Italy while Lady Arundel left for the Low Countries. He lived most in Padua, but also visited Parma. Sadly, Arundel’s eldest grandson was now a lunatic and another grandson had become a Dominican monk; he was also angry that his wife had “scattered” his collection. The Arundel sons failed to sell their father’s art, the best items having been sent abroad to avoid looting. However, the Spanish Ambassador in London had his eye on Arundel’s impressive Raphael (Pope Leo X with his Cardinals). It was obtained and sent to Spain where Velasquez pronounced it a copy as the cardinal in the background differed from Rossi. It is now thought to be a third version painted by Bugiardini (Rome, Galleria Corsini) for Cardinal Cibo who is substituted for de’ Rossi. In 1654 Lady Arundel died in Amsterdam, just two years after her eldest son – Lord Maltravers (1608-1652). They quarrelled over Arundel’s inheritance and her Catholic faith – so she left the collection to her younger son, Lord Stafford (1612-1680). Stafford was also a Catholic, and he lost no time in selling his inheritance. Amongst the pictures to go were Veronese’s Christ and the Centurion. At this stage Lady Arundel’s will was contested by the son of her eldest son, so eventually Lord Stafford and his nephew compromised by dividing the pictures between them. Some of the Arundels were brought back to England, e.g. Holbein’s Portrait of Erasmus (NG, on loan from Longford Castle Collection). John Evelyn was scathing about the dispersal of Arundel’s collection. Most of Arundel’s pictures remained in Amsterdam for the next thirty years until they were finally dispersed by auction in 1684. 
 Francis Haskell describes the Arundel holdings and their fate: “One gets the impression of a sort of incredible emporium, owned by absentee shareholders, which, over the years, was dipped into by purchasers of all kinds, who presumably paid their bills of exchange into the accounts of the various family members who had a stake in what remained. The name of Arundel provided a plausible guarantee of quality and authenticity, but who made the arrangements, and who determined the price is not at all clear.” The King’s Pictures, 113-114.» Kings, Collectors, and Paintings in the 17th Century: Week 2 : The Arundel and Buckingham Collections
From Wikipedia again: Two grandchildren claimed half of the inheritance and sent Sir Edward Walker to the Netherlands. In 1655 Stafford was arrested in Utrecht, but released within a few weeks. Henry Howard, 6th Duke of Norfolk and his brother Charles were keen on getting the paintings and went in Utrecht to court in 1658 and 1661.
«Before his relations could interfere Lord Stafford had sold a number of pictures to the Spanish Ambassador in London, to Eberhard Jabach, of Cologne, and to the agent of the Archduke Leopold, and this may account for the fact that certain of them remained abroad, such as the Jane Seymour and Dr. Chamber in Vienna, and the Thomas and John Godsalve in Dresden.» Hans Holbein the Younger by Arthur Bensley Chamberlain
The Three Brothers Browne, by Isaac Oliver, signed with monogram, inscribed and dated 1598, which is in all likelihood was in the Arundel collection, is today at Burghley House, though this does not appear to have been a part of the bequest from the Countess of Devonshire.
There was another miniature by Oliver included in the bequest in addition to Venus and Cupid, that of Robert Devereux, 2nd Earl of Essex, by Isaac Oliver, circa 1600, but this does not appear to have been in the Arundel collection.
The reason this is interesting, is that after discovering the Duckett portrait, I went through the inventory of the Arundel collection with an eye for a woman not identified as Lady Jane Grey, but with a description fitting with the apperance of the lady in the Duckett portrait, to see if we might trace the Lumley portrait of Lady Jane Grey that way.
Since it seems I was the first to connect the Duckett portrait to Lady Jane Grey, I assumed that had not been done before.
I did not find that (and upon further reflection, I realised that the presence of a cartellino certainly would have told the takers of the inventory who she was. After all, how else did they know who all of these obscure English 'celebrities' from the 1500's were in a foreign country one hundred years after their deaths?)
I did, however, find the following entries:
448. Portrait of a Woman, in profile.
452. Portrait of a Woman in a small black cap with a white plume.
457. A Lady in a cap and plume.
See No. 452.
591. A Woman with a small cap and plume.
See No. 452
706. Portrait without inscription.
714. Joan Shorr, advocate.
761. A Woman to the waist.
775. Two Portraits, one of a Man, the other of a Woman, in a box. Wood.
Again, this is proof of absolutely nothing, but the descriptions in these entries do correspond perfectly with the Anglesey Abbey portrait, the little miniature, and the Ansty Hall miniature. Furthermore, something about the Mary Frederica Sophia Hervey writes about these entries, makes it seem as if she sometimes wonders if the different entries sometimes describe the same item. The entries were after all created Mary Frederica Sophia Hervey, not the compilers, from a «far from clear inventory». She undertook the task of trying to make some sense of it, and from the remarks of everyone who has seen the original inventory, it seems as if we owe her a great thanks for that.
Even if we put these descriptions together, we get:
Portrait without inscription of a Woman in a small black cap with a white plume, in profile, to the waist.
Which fits perfectly with the Anglesey Abbey portrait. In fact, it is the very words I would have used if I were to try to describe the Anglesey Abbey Portrait in a list or to a person who was unfamiliar with it.
The entry 775. Two Portraits, one of a Man, the other of a Woman, in a box. Wood. will be become important later on.
However, if the Chatsworth Portrait took a little detour to the Continent, the Cavendishes must have retrieved it.
Because I do believe that this, the Anglesey Abbey portrait, was the portrait of Lady Jane Grey that hung on the bedroom wall of Bess of Hardwick in 1560.
If it had indeed travelled with the Arundels away from the civil war, its very entries in the less auspicious part of the inventory makes it clear that it would probably not have been amongst the most coveted part of the Arundel collection, allowing someone who really wanted it to discreetly purchase it rather than it being dispersed before anyone had got their bearings like some of the more prestigious items in the collection.
Because a portrait of Lady Jane Grey does keep appearing and re-appearing intermittently at Chatsworth and other Duke of Devonshire properties in the following centuries.
«The travel diarist John Byng, Viscount Torrington, did report a portrait of Jane "much neglected" and hanging in the Great Drawing Room when he visited Hardwick Hall in 1789.» A Queen of a New Invention, by J. Stephan Edwards, p. 184
The «portrait in the Middle Drawing Room that was attributed to the seventeenth-century artist Anthony van Dyck and curiously said to depict "the Dutchess[sic] of Suffolk and Lady Jane Grey."» mentioned by the Devonshire House inventorist in 1811 was probably the portrait shown above of Christian, Lady Cavendish, with her daughter.
«At least nine portraits of women that could no longer be identified were recorded [at Hardwick] in 1811. One full-length was suspected to depict Arbella Stuart, while a half-length was said to be of "a Lady supposed to be Queen" The other seven were entirely beyond recognition. And a further "Twenty other Old Portraits and Paintings various, [were] very much defaced and Bad."» A Queen of a New Invention, by J. Stephan Edwards, p. 184
«A portrait of Jane Grey reappeared briefly at Hardwick Hall early in the nineteenth century.» A Queen of a New Invention, by J. Stephan Edwards, p. 185
«But it had disappeared again by 1860, when Lady Louisa Cavendish Egerton compiled a catalogue of the pictures at Hardwick Hall. She listed just over 300 pictures, none of which were said to depict Jane Grey. Four were of unidentified women, two of which were dated to the late-seventeenth century, and a third had been newly acquired in the nineteenth century. The fourth was not described. Five years later, Sir George Scharf, Director of the National Portrait Gallery, surveyed over 300 pictures at Hardwick Hall, though he offered descriptions of the content of only 261, including over a dozen identified using the single word "Unknown." Then in 1903, Cecil Foljambe, Lord Hawkesbury created a printed catalogue of the collection at Hardwick Hall. Foljambe counted "four curious paintings on panels, supposed to have come from the old Hall," though he did not record their content. Neither did he find any picture that could today be identified as a potential depiction of Jane Grey.» A Queen of a New Invention, by J. Stephan Edwards, p. 185
«In all likelihood, Bess of Hardwick's authentic portrait of Jane Grey was one of the many found to be "much defaced" or "bad" late in the eighteenth century and thus not worth preserving. The picture was almost certainly discarded or deliberately destroyed before the middle of the nineteenth century.» A Queen of a New Invention, by J. Stephan Edwards, p. 185
While this is a reasonable assumption to make, I simply do not believe that they had them destroyed.
There is nothing about the Devonshire-Cavendishes and their 500-year-old manor filled with 300-year-old stuff that indicate that they are a particularly unsentimental lot.
If you look around in your own closets, chests of drawers and attics, how much of it is stuff you will never have use for again?
How much of it are you willing to throw away?
The Devonshire-Cavendishes didn't have to throw away anything that they did not particularly want to.
Presented with a list from some underling about which of their possessions that person thought useless, did they go, «Burn them all, Jeeves!!!» or did they simply shove the old, derelict paintings into one of the 300 rooms they were not currently occupying or piled them up in a corner of their 1000 square feet attic that was currently not in use?
Also, We Do Not Destroy Paintings must have been a particularly strong more in the aristocracy and otherwise, considering all of the paintings of, shall we say, varying quality that have survived through the ages and have been passed down to us.
After all, if they wanted to be rid of one or many, there was always someone willing to buy it, or some fringe family member eager for an heirloom.
The portrait of Lady Jane Grey was said to be in a poor condition indeed.
J. Stephan Edwards offers an excellent explanation for how this may have come to be: «In the instance of the Chatsworth Portrait owned by Bess of Hardwick in 1559, the evidence suggests that the portrait suffered significant decay over time, either through conscious neglect or natural processes. Most habitable rooms in pre-modern houses included a fireplace, and those fireplaces often discharged some measure of smoke into the room itself. As is the case with modern households in which the residents smoke tobacco products, smoke residue could and did accumulate over time, eventually obscuring the image. Inventories taken in the nineteenth century at the houses of Bess’s descendants, Chatsworth House and Hardwick Hall, revealed over two dozen portraits in which the image was entirely obscured by soot and dirt. That soot and dirt also often caused chemical reactions in the protective varnish, the paintwork itself, or even the supporting wood panel, especially in those instances when the panel became wet for some reason, e.g.: ‘rising damp,’ flooding, leaking roofs. Panels became warped, split, or riddled with wood worm.» Queen Jane, Where Are You? – Some Grey Matter
If my theory is right, the additions of an amateur family painter in the early 1600's probably didn't help matters, either.
So, you have a derelict painting. At some point, you will probably want to have something done with it, yes.
It is my belief that they brought the picture of Lady Jane Grey with them to London.
J. Stephan Edwards describes how the portrait of Lady Jane Grey was moved around between Chatsworth and Old Hardwick Hall and New Hardwick Hall as the Cavendishes built, rebuilt, and redecorated.
The interesting thing is that all of these residences ended up with the same of Bess's children.
Chatsworth House – Bess died in 1608 and Chatsworth was passed to her eldest son, Henry. The estate was purchased from Henry by his brother William Cavendish, 1st Earl of Devonshire, for £10,000.
Hardwick Hall (Old and New) – After Bess's death in 1608, the house passed to her son William Cavendish, 1st Earl of Devonshire. His great-grandson, William, was created 1st Duke of Devonshire in 1694. The Devonshires made another of Bess's great houses, Chatsworth, their principal seat.
And so William Cavendish, 1st Earl of Devonshire, becomes the third of Bess of Hardwick children to be of interest to us.
Of course the above residences were in no way enough. A London house to be fashionable for the season was just the thing. Enter: Devonshire House.
Following a fire in 1733 it was rebuilt for William Cavendish, 3rd Duke of Devonshire, in the Palladian style.
Since we have observations of the portrait of Lady Jane Grey after 1733, I think we can safely assume that it was not consumed by flames that year.
But I do believe that Devonshire House is where it ended up.
Devonshire House was completed about 1740, it stood empty after the First World War and was demolished in 1924.
Quoting freely from Wikipedia: Following World War I, many aristocratic families gave up their London houses, and Devonshire House was no exception; it was deserted in 1919.
The reason for abandonment was that the 9th Duke was the first of his family to have to pay death duties; these amounted to over £500,000. Additionally, he inherited the debts of the 7th Duke. This double burden required the sale of many of the family's valuables, including books printed by William Caxton, many Shakespeare 1st editions, and Devonshire House with its even more valuable three acres of gardens. The sale was finalised in 1920, for a price of £750,000, and the house demolished.
Some of the paintings and furniture are now at the Devonshire principal seat, Chatsworth House.
The above portrait of Alice Spencer is very similar to another one of Lady Harington: https://www.tate.org.uk/art/artworks/gheeraerts-portrait-of-mary-rogers-lady-harington-t01872 Faces look identical
I completely agree. The likeness is startling.