The early logical and mathematical work of Bertrand Russell and Ludwig Wittgenstein was made possible by Gottlob Frege (1848 - 1925) who taught at the University of Jena. In 1893 a Jena publisher brought out the first volume of his pioneering Basic Laws of Arithmetic and the second volume in 1903. Sales of the first volume had been so poor that Frege had to pay for publication of the second. But he did not self-publish and nor did he, in the ordinary sense, vanity publish since his publisher Hermann Pohle exercised discretion in what he published - and Pohle could point to the fact that on the title page Frege would be identified by his academic titles (as he was) and claim that as adequate justification for publishing a work which neither Pohle or anyone else understood.
The conventional way of characterising what happened is to say that Frege was published on commission. Under such arrangements, the author pays all the printing costs up-front; the publisher retains a commission on all the income from sales, but hands over the balance to an author who may or may not recover their original outlay. Both Jane Austen and Lewis Carroll were published on commission. Where copyright law exists, it is likely that the author published this way will retain copyright. In contrast, when a publisher bears all the costs, they sometimes agree to do it only if copyright passes to them. They are, after all, taking all the risk. Academic journals, for example, took their pick of submitted articles, paid the whole cost of publishing, took the copyright, and paid no royalties. Like all academics of the pre-internet period, I signed up for that many times. Now I observe with curiosity the fact that downloads of things I wrote are on sale from publishers who took everything and paid nothing - though it’s true, neither authors or publishers foresaw a world in which something called a download might exist let alone be sold.
Permutations on these arrangements are easily imaginable and modern printing and publishing technologies have expanded the range of possibilities, once ideologically polarised (by proper publishers - MRDA) into proper publishing and vanity publishing.
But it was never really that simple, as the practice of publishing on commission illustrates, and it still isn’t simple. Self-publishers are avoiding the older, rather quaint vanity publishing firms. Publishers can apply to the Arts Council for subsidies to publish books which are unlikely to make a profit, perhaps because of the very significant cost of getting a good translation made. Universities may subsidise publication of a faculty member’s book if something like the need for colour illustrations would otherwise push up the cover price to levels which would deter everyone except librarians. And so on.
The quality control which a publisher’s editors like to think they exercise always takes place in a context where the balance sheet has to be considered. A subsidy - doesn’t matter where it comes from - can alter the equation and make the difference between acceptance and rejection. Of course, there are always other factors in play. Modern corporate publishing has become so competitive, and margins so tight, that I very much doubt that editors can afford to devote more than a very few hours to reading, judging, and seeking to improve the books they commission. My guess is that nowadays a large proportion are nodded through on the basis of the author’s previous track record. They are published unread.