Links and Notes for April 30th, 2021
Published by marco on
Below are links to articles, highlighted passages, and occasional annotations for the week ending on the date in the title, enriching the raw data from Instapaper Likes and Twitter. They are intentionally succinct, else they’d be articles and probably end up in the gigantic backlog of unpublished drafts. YMMV.
“[…] events in Chile confirm the dangers of encouraging any complacency. Despite undertaking one of the fastest vaccine programmes (next to Israel and the UK) with some 40 percent of its population receiving at least one dose of a coronavirus vaccine, the number of daily cases in the country is higher than at any time in the last year.
“The use of vaccination to justify reducing social distancing measures played a key role in Chile’s outbreak. But it is social inequality that is the major factor. A research study into COVID-19 incidences in Chile’s capital, Santiago, for Science —the journal for the American Association for the Advancement of Science—records a “strong association between socioeconomic status and both COVID-19 outcomes and public health capacity.””
The West is Practicing Vaccine Apartheid at a Global Level by Prabir Purkayastha (CounterPunch)
“How much of the vaccine supplies from Sinovac, a Beijing-based biopharmaceutical company, and Sinopharm, a Chinese state-owned company, was administered locally in China, and how much has been provided to the rest of the world? About 115 million doses have been used in China, and the same amount has gone to the rest of the world, according to an April 5 article in Nikkei Asia, which relied on data provided by Airfinity, an analytics company. Similarly, based on the figures released by India’s Ministry of External Affairs website on April 15, 2021, more than 65 million doses of the Serum Institute’s Covishield vaccine—licensed from AstraZeneca—have been exported to other countries.”
“China and India are the only two major countries that have been willing to export vaccines while also vaccinating their own people.”
“Apart from the intellectual property rights issue, the major roadblock to quickly ramping up global vaccine production is that the rich countries—the United States, the EU, and the UK—have been refusing to export not only vaccines but also the supplies of intermediate products and raw materials required for vaccine production in other countries.”
“Data of Brazil’s Sinovac’s CoronaVac trials showed that it provided 78 percent protection in mild cases and 100 percent protection in moderate and severe cases, according to an article in Bloomberg. Esper Kallas from the School of Medicine, University of São Paulo, Brazil, pointed out in an article in Science Magazine, “If you can prevent someone being seen by a doctor by 78 percent and prevent hospital admissions by 100 percent, let’s give a toast and celebrate.””
“I have earlier reported about the World Trade Organization rules and the rich countries’ unwillingness to temporarily suspend intellectual property rights rules so that all the vaccine producers can re-engineer their facilities very quickly to produce COVID-19 vaccines. In the books of the rich countries, the tens of billions of dollars to be earned as profits in the vaccine market by Big Pharma far outweigh the benefits of saving millions of lives. This also explains the vicious campaign against Chinese and Russian vaccines.”
Moderna’s Pledge Not to Enforce the Patents on Their COVID-19 Vaccine Is Worthless by Alexander Zaitchik (Jacobin)
“In the age of undisclosed information, applicants are no longer required to provide governments with meaningful collateral in exchange for the benefits of government-protected monopolies. Instead, they can provide partial maps to technologies they have no intention of revealing in full — fragments designed to frustrate, obfuscate, and occlude, providing knowledge that’s necessary but not sufficient to actually make the thing.”
“This “invulnerable exclusivity” is harmless enough when it protects secret soda formulas and hamburger mystery sauces. It’s less cute when it blocks countries from using their legal right to manufacture and import lifesaving medicines. But that is exactly the kind of activity the new IP regime was designed to frustrate.”
“In the context of the WTO, the adoption of US trade secret policy has major implications for the nations’ legal right to issue compulsory licenses, a right confirmed in 2003 at the WTO Doha conference. If key aspects of a complex biopharmaceutical process are kept behind padlocked silos, then you might as well make a paper airplane with the actual product patent.”
“Historically, compulsory licensing has been deployed as a threat more than an actual weapon. When a government has all the information it needs to begin generic production of a drug, it has a decent chance of forcing the patent owner to join them at the negotiating table, giving both sides a chance to hammer out a compromise.”
“The incredible shrinking utility of the traditional patent is a modern feature of all high-technology fields. Around 80 percent of license agreements now include technology transfer clauses covering trade secrets and other forms of undisclosed knowledge. The international IP consultant Robert Sherwood calls trade secrets the “workhorse of technology transfer.” More and more, you can’t build anything without trade secrets, and no policy tool short of a federal police raid can force companies to give them up.”
“If the scientists clear these hurdles, they must then produce a functioning manufacturing design without access to hundreds and possibly thousands of trade secrets and pieces of technical know-how. Before the rise of trade secrets, patents were required to include all information related to the product’s “best production method.” Now, patentees are allowed to meet a much lower standard for production methods; it can keep the details of the “best method” secret until the end of time.”
““In the 1990s, patents were considered the primary puzzle for access-to-medicines advocates to solve,” says Christopher Morten, a patent lawyer who used to represent the pharmaceutical industry and now teaches at NYU. “Today, it’s proprietary clinical trial data, regulatory exclusivities, and other forms of non-patent exclusivity.””
Economy & Finance
“If you are a company — particularly a Chinese company — that would like to be publicly traded in the U.S., but that would prefer to avoid the scrutiny that comes with an initial public offering, doing a “reverse merger” — acquiring the empty shell of a near-defunct but public U.S. company — is often an easy way to do it. A deli with $13,976 of sales, but with careful and pristine SEC filings, might be rather valuable to a certain sort of Macau- or Hong Kong-based investor.”
“Still it is strange to characterize this as primarily securities fraud against the shareholders, to think that the problem here was lying. The problem is not that Credit Suisse went around telling shareholders “we try not to lose money on dumb stuff” but had a secret undisclosed nefarious plan to lose money on dumb stuff. The problem is that Credit Suisse tried not to lose money on dumb stuff and failed.”
“But if I want you not to cut down trees for my carbon capture program, it is harder to measure how many trees you didn’t cut down. Just sitting here right now, typing this column on my computer, I have cut down zero trees, which means in theory that there are absolutely billions of trees that I have not cut down. Where is my check? A landowner might have planned to cut down only a few trees this year, but she will have incentives to say “I was planning to cut down all my trees,” in order to get paid for not cutting down all of them. She might have trees that are impossibly un-economic to cut down, but it’s easy enough for her not to cut them down.”
“If I agree to pay you not to cut down 100 trees, though, what’s to stop you from getting paid by someone else not to cut down the same trees? The trees stay there; you can sell the concept of them staying there as many times as you like.”
“If you are smart enough to avoid all the dumb trades, the people with the borderline trades — the risky hairy trades, the trades that might well be dumb, but that, if they’re not dumb, will be very lucrative — will go elsewhere. Why deal with you and your strict culture of risk management, when they could deal with someone else’s nonexistent culture of risk management?”
“Credit Suisse failed to protect itself from its Archegos exposure in part because it had not yet instituted a system that monitored in real time how much risk a position created for the bank as the prices of the underlying securities changed,”
That seems like kind of a gross oversight for a bank backstopping billions of dollars of swaps.
“Public disclosures showed the various banks that Mr. Hwang had worked with—including Credit Suisse Group AG, Nomura Holdings Inc., Morgan Stanley and Goldman Sachs Group Inc.—as being the major shareholders in stocks rather than Archegos. When a bank is listed as a shareholder, companies have no way of knowing if the investment is on behalf of a single investor, multiple investors or the bank itself.”
“But in general if a company sees a handful of big banks buying up positions in its stock, it is going to worry that they are doing so on behalf of a swaps customer, and that the swaps customer is an activist. And they’d prefer to know as much as possible as soon as possible.”
“You promise not to cut down the trees in a conservation agreement (and get some tax benefit or something), and then you promise not to cut them down in a forest offset agreement (and get paid); you try to squeeze as much juice as you can out of not cutting down the same trees.”
“[…] the benefit that the buyers get is not “lumber to build stuff with” but rather “good publicity and credit with sustainability ratings firms,” their incentives to check are somewhat attenuated. If you sell not cutting down the same trees twice, both buyers get pretty much the same benefit as if you’d only sold them once.”
“The point is that (1) any platform for selling not-cutting-down-trees will not let you sell the same trees twice, but (2) to the extent there are competing standards and service providers, they may not communicate perfectly with each other, and you can maybe sell the same trees twice in two different places.”
“I do not want to give you investing advice, but I will say that if you went out and spent $100 million to buy all of the stock of Hometown International — which, again, is a deli — you would end up owning only about 5% of the company. I … I would not personally do that trade? But obviously you do what you want. We are way past my ability to advise here. Look at me, doing math, like an absolute chump.”
“See, Bitcoin and Ether have no exposure to the Shiba Inu factor, that’s your problem right there. “I see a lot of support at the $0.42 and $0.69 levels,” some Dogecoin technical analyst is going to have to write, “because those are the good joke numbers.” Actually I guess she’ll write “Such 4/20, many support, so resistance, much wow, how Fibonacci.” Do people even remember Doge, or has the coin totally replaced the meme?”
“We talked above, and yesterday, about the deli. People find the deli’s $100 million, or $2 billion, valuation very strange. But the deli is real! Dogecoin is explicitly a joke and it’s worth more than Ford Motor Co., because it’s 4/20 today. And you’re going to go to work today and build a discounted cash flow model for some industrial company and try to figure out its debt capacity and how to market it to potential strategic buyers, as though any of those things were things. I will be disappointed if Elon Musk doesn’t announce that he’s buying GameStop Corp. today. Or the deli.”
“If you buy a binary option from a scammer, he is not selling you some option product that he bought on an exchange. He is just taking the other side of the bet with you. And you can’t compare his prices to those of a normal bank or broker, or see the correct odds for the bet. So he can give you the incorrect odds, to make sure that you will usually lose. If he does this with enough people, he is just running a casino with a large house edge, and he can get rich.”
Public Policy & Politics
“Während zehntausende Menschen ihr Leben verloren, hunderttausende schwere Schäden erlitten und Millionen Teile ihres Einkommens einbüßten, haben sich die oberen zehn Prozent der Gesellschaft pervers bereichert. Mit ihrer Öffnungspolitik geht die herrschende Elite über Leichen, um ihre Profite zu sichern.”
“Sie kritisiert nicht, dass die Arbeiter durch die Identitätspolitik gespalten und von einem gemeinsamen Kampf gegen die kapitalistische Klassengesellschaft abgehalten werden. Sondern sie unterstützt diese Spaltung, indem sie sich für die Aussonderung von Zuwanderern und ihre Diskriminierung auf dem Arbeitsmarkt ausspricht. So will sie von der Verantwortung der Linkspartei ablenken.”
“All diese haltlosen Behauptungen sind von Corona-Leugnern hinlänglich bekannt und von Wissenschaftlern längst widerlegt. Würde man die von Wagenknecht propagierte Durchseuchung der Bevölkerung unter 65 Jahren zulassen, stünden in den unteren Altersklassen bis zu 180.000 Menschenleben auf dem Spiel, errechnete etwa die Virologin Melanie Brinkmann. Für Wagenknecht ist es dieses Opfer offenbar wert, „damit wir unsere Wirtschaft nicht ruinieren“, wie sie im Februar bei Anne Will erklärte.”
“The White House openly admits that during their first phone call on Good Friday, Biden offered Zelensky, “unwavering support for Ukraine sovereignty and territorial integrity in the face of Russia’s ongoing aggression in Donbass and Crimea.” For those of you who don’t speak fluent NATO, that’s code for, ‘Bomb! Bomb! Bomb!’”
“We lost patience with simply perverting democracy in 2014 when the democratically elected president Viktor Yanukovych chose closer ties with Russia over Euro servitude. The US called in its allies in the nation’s fledgling neo-Nazi movement and they launched a coup that chased that nation’s democracy to Moscow and ended with an openly racist junta in power in Kiev.”
“The western media blames Russia for every mortar out of Mariupol, though they’ve offered absolutely bupkis in the way of evidence for their cockamamie conspiracy theory of a Russian invasion. If Russia had invaded, this thing would have ended like Georgia’s 2008 attempt to purge itself of ethnic undesirables for NATO membership. Russia kicked their ass back to Tbilisi before John McCain could get his dick hard.”
“Finally, after years of abuse from Kiev, the last straw fell with the coup against Yanukovych, whom the region had voted for by 90%. When the same fascist flags their fathers died beneath went up over Kiev and announcements were made to ban their indigenous Russian language, the Novorossiyans said enough.”
The Director of National Intelligence’s Hypocritical Annual Threat Assessment by Maj. Danny Sjursen (ScheerPost)
“Worse still is the list of predicted “tools” the report’s authors “assess” Moscow will employ “to advance its agenda and undermine the United States:” …especially influence campaigns, intelligence and counterterrorism cooperation, military aid and combined exercises, mercenary operations, assassinations, and arms sales… Seriously, has Washington’s repeatedly demonstrated unwillingness or inability to [accurately] see its own reflection in the mirror – thereby outing itself as a member of the declining hegemonic undead – even been more obvious than in this list of predicted nasty acts planned by America’s favored naughtiest nemesis? Tell me: which of these enumerated nefarious tactics is the USnot also doing, and usually doing more of?”
“(Recall that it was nearly 50 years ago that Air Force General Curtis LeMay, head of the Strategic Air Command, interrupted a junior briefing officer who had repeatedly referred to the Soviet Union as “the enemy.” LeMay supposedly corrected him thus: “Young man, the Soviet Union is our adversary. Our enemy is the Navy.”)”
The Washington Post’s phony campaign on Uyghur “genocide” by Peter Symonds (WSWS)
“The rapidly escalating US-led campaign on “Uyghur genocide” recalls the “big lie” exploited by the US and its NATO allies to initiate the murderous bombing of Serbia in 1999. The Clinton administration justified its “humanitarian intervention” as a mission to prevent the massacre of Kosovo’s ethnic Albanian population by the new “Hitler”—Serbian leader Slobodan Milosevic. A thoroughly pliant and corrupt American and international media immediately fell into line with sensational stories of Serbian atrocities.”
“Lurid claims that 100,000 ethnic Albanians had been slaughtered proved utterly false in the aftermath of the war. The actual death toll was around 2,000 and most of those killings were committed by the armed separatist group, the Kosovo Liberation Army (KLA). Washington had previously branded the KLA as a terrorist organisation due to its ties with Al Qaeda, but rapidly reversed course, provided money and arms, and declared it to be the sole legitimate representative of Kosovo’s population. KLA head Hashim Thaçi, who became the head of the US-backed Kosovo mini-state, is currently facing charges of war crimes in the Hague.”
“The outrage over Navalny’s imprisonment and resulting health crisis is an object lesson in imperialist cynicism and intrigue. Those most passionately invoking his democratic rights are the architects of WikiLeaks founder Julian Assange’s continuing and vastly more severe persecution.”
“Assange is a heroic journalist who played a leading role in the exposure of some of the worst imperialist crimes of the 21st century, from covered-up details of the brutal occupation of Iraq and Afghanistan to US torture camps and extraordinary renditions. Navalny is a right-wing, nationalist politician, who has referred to migrants from the Caucasus as “cockroaches” that should be killed. He represents a wing of the Russian oligarchy opposed to President Vladimir Putin and in favour of opening Russia up more widely to Western imperialism. It is this difference which underpins their night and day treatment.”
“Johnson praised the “brave” Navalny and said the Russian ruling “fails to meet the most basic standards of justice.” US national security adviser Jake Sullivan condemned the “violation of human rights.” Merkel declared the Navalny verdict “far from any rule of law standards.” French President Emmanuel Macron stated, “respect for human rights such as democratic freedom are not negotiable.””
Unsurprising hypocrisy from all of them.
“Amnesty International refused to acknowledge Assange as a prisoner of conscience for years but was so quick to apply the label to Navalny that they were forced into an embarrassing retreat in acknowledgment of his record of “hate speech” a few months later.”
“US Democratic Party Senator Bernie Sanders has maintained near total silence on Assange, issuing a single tweet opposing his indictment in May 2019 that succeeded in not mentioning the WikiLeaks founder by name. He tweeted this Monday: “Make no mistake about what is happening here: activist Aleksei Navalny is being murdered in front of the world by Vladimir Putin for the crime of exposing Putin’s vast corruption. Navalny’s doctors must be allowed to see him immediately.””
Jesus Christ on a crutch, Bernie, you can be such a fucking shill sometimes.
“The massive allocation of money and resources to the U.S. military, which includes Biden’s request for $715 billion for the Defense Department in fiscal year 2022, a $11.3 billion, or 1.6 percent increase, over 2021, is not in the end about national defense. The bloated military budget is designed, as Seymour Melman explained in his book, “The Permanent War Economy,” primarily to keep the American economy from collapsing. All we really make anymore are weapons.”
“The flowery rhetoric used to justify the subjugation of other nations so corporations can plunder natural resources and exploit cheap labor is solely for domestic consumption. The generals, intelligence operatives, diplomats, bankers and corporate executives that manage empire find this idealistic talk risible.”
“The loss of the dollar as the global reserve currency will probably mark the final chapter of the American empire. In 2015, the dollar accounted for 90 percent of bilateral transactions between China and Russia, a percentage that has since fallen to about 50 percent. The use of sanctions as a weapon against China and Russia pushes these countries to replace the dollar with their own national currencies. Russia, as part of this move away from the dollar, has begun accumulating yuan reserves.”
“The meaninglessness of President Biden’s announcement becomes apparent when we consider that the Pentagon employs more than seven contractors for every serviceman or woman in Afghanistan, an increase from one contractor for every serviceman or woman a decade ago. As of January, more than 18,000 contractors remained in Afghanistan, according to a Defense Department report, when official troop totals had been reduced to 2,500.”
“[…] the current Afghan government led by Ashraf Ghani is largely a creation of the United States. Its military is funded by the United States at a cost of around $4 billion per year. This support is going to continue — unless Congress cuts it off — alongside large-scale U.S. foreign aid programs that amount to nearly $1 billion per year.”
Smithfield Foods by Jane McAlevey
“There were so many employer-inflicted casualties in this particular class war, the rather stunning fact that 2,000 individuals lost their jobs in a single day because they had wanted a union can almost get buried in the long list of other outrages. That they chose to leave by engaging in a massive wildcat strike that would hurt the boss, if only for several days, speaks to their deep sense of human dignity, and their bravery. By this time, there were almost daily daring actions by workers on the inside and vicious responses from the employer, and the fight was shifting outside, where it would generate more support.”
“Most liberals, including those in the U.S. mainstream media, readily understand that when a repressive regime somewhere in the world calls for an election to add a fig leaf of legitimacy to its continued rule, the election is in no way free or fair. Yet these same people cannot seem to grasp that an employer like Smithfield, which effectively got rid of 2,000 immigrant workers (pro-union voters)—many of them encouraged by the company to come to the U.S. illegally in the first place—and was systematically driving a race war inside the plant, is not likely to hold a “free and fair” union election.”
“On the other, you also can’t just allege wrongdoing because you don’t like the result. If independent auditors verified it, tread lightly unless you have rock-solid proof.”
“The workers in the poultry plant, who could be my kids in the future, they drive two hours a day to earn $250 per week with no health insurance, and the company is building a $5 million expansion in their plant.””
“Ironically, the length of time it took workers to get to work, and to where the employer stationed the time clock, would become an issue in the first contract negotiations. The Livestock workers won the right to a parking lot in the back, and saved over one hour each day of walking the plant in unpaid status. These same workers now have the legal right to walk through their own factory anytime, often in paid status, to conduct union building efforts post–contract settlement.”
Behind the News, 4/29/21: Ben Burgis on Cancel Culture by Doug Henwood
“Ben Burgis: I think there is a interest in a kind of of performative contradiction there … because surely anyone who identifies with left politics thinks that the United States is kind of drunk on incarceration, that we should have a vastly less carceral, criminal-justice system. For example, that we should “ban the box” … that box on employment applications that asks if you’ve been convicted of a felony, so that ex-convicts will be able to get regular, gainful, non-criminal employment when they get out of prison. There’s something very funny going on here: in the views of people who, on the one hand, presumably believe that rapists and murderers should get a second chance, but, apparently, don’t believe that people who have a history of bad tweets should get a second chance.
“Doug Henwood: I’m reminded of something that Michael Kinsley said decades ago: The right is always looking for converts, while the left is always looking for heretics.”
Unfortunately, no. The market has decided that those things don’t matter. If Amazon makes a ton of money without having trouble employing people even though it treats them poorly, what is wrong with their model? What is going to stop them from doing what we consider to be criminal if it’s not actually criminal? Their own morality? Their conscience? Their desire to limit their profits in exchange for a better reputation? None of these things are true. They will take the money and the fact that there is no downside to their business practices. The economy optimizes toward this because it’s allowed. Change the incentives.
This is an excellent interview with anti-war activist David Harris. He opposed one of the greatest crimes of the 20th century: the Vietnam War.
It is becoming impossible to escape this kind of garbage.
“The finding feeds into a persistent idea that Russia is behind the attacks and is carrying them out with some sort of clandestine, portable, microwave-based weapon. Since the cases came to light in Cuba, US personnel have also reported similar incidents while in Russia. Defense officials told CNN that Russia is among the top suspects but that China is included as well.”
Sure. Why not? While you’re just making wild, unanswerable, and largely risk-free allegations, just smear everyone you can think of.
I heard they’ve spotted bat boy again as well.
“Everyone knows what you mean when you say something is “trending,” and everyone has a vague understanding that the phenomenon is generally bad, but for whatever reason it goes unrecognized that this keeps happening because Twitter lets you slap a hashtag in front of something, scoop up a few thousand loons, and get your opinion fast-tracked to the top tier of the collective twitter consciousness. Twitter absolutely knows that this happens but they have their own set of incentives to satisfy and the hashtag is their brand, so they’ve basically baked this dysfunction into their user experience.”
“Tom: I tell you what. Today I’ve been depressed. I was tryin’ to buy a car, you know, and I got turned down at the dealership, for financing, […] and it just struck me, like, how … it’s just dehumanizing to, like, not be rich in this country. Even if you’re just, like, I do fine, you know what I mean? I’m like comparatively wealthy, compared to most, you know what I mean? […] Anyway, I’m just bitchin’ about how hard it is to buy things and navigate this world.
“And then, like, I go to the group chat of my buddies from college. They’re all like pissin’ and moain’ of, like, the specter of, like, Biden’s 40% capital-gains tax—which I, like, don’t think is gonna happen—but it’s, like, bro, you got motherfuckers out here, fuckin’ payin’ taxes on goddamned unemployment—which is 60% or whatever it is, of what they were making—and y’all are pissin’ and moanin’ because ya made $280 grand and not $400 grand? Like suck the biggest fuckin’ dick that exists.”
“Tanya: If I were a betting girl boss, I would have bet that we would get universal health-care out of a pandemic. I actually thought that, at one point. That’s how naive I am. When the WHO said the #1 thing you can do to curb this pandemic is universal health care…and we withdrew from the WHO in response, I don’t think my mind’s been right since, honestly. That was a pretty big turning point for me, to be honest.”
Art & Literature
“But now they’re betting on all of us. And they’re not just betting when they think our scores will go up; they’re also trying to short us, to make money off of an anticipated decline in our reputational ranking. It’s out of control!”
“The viewer stats on my government streaming page were off the charts, and the betting activity on the new site was as well. I was like some Hindu god, whose every smallest whim creates and destroys entire worlds: I scratch my head, and a thousand mortals lay down their money on me; I cough, and two thousand take theirs back. I stretch my arms at daybreak and men harvest their coins. Just by living I am a source of wealth. I am worth something.”
“Anyhow the betting site is going absolutely crazy. There are derivative markets for betting on when I go live again, and on how soon after I go live I’ll make my first outrage face, and on how much my score will move up or down after I make my first outrage face, and on what sort of face I will make when my webcam catches me noticing how much my score has gone up or down after I’ve made my first outrage face. There are more markets still, but the truth is I lose my power of concentration when I try to read much further.”
Philosophy & Sociology
“There’s that anecdote of a neuroscientist and a physicist talking and the physicist says ‘the most important problem in science is what’s the origin of the universe?’ and the neuroscientist says ‘what organ led you to that decision?’”
Query Engines: Push vs. Pull by Justin Affray
“The other tricky thing with DAGs in a pull-based model is that an operator in such a system is at the mercy of its downstream operators: a row that might be read in the future by any of its consumers must be kept around (or must be able to be re-computed). One general solution to this is for an operator to buffer all of its rows that get output so you can re-hand them out, but introducing potentially unbounded buffering at every operator boundary is undesirable (but is, by necessity, what Postgres and CockroachDB do for WITH having multiple consumers).”
“This is why in a streaming system like Flink or Materialize you’ll typically see push-based systems used: the inputs to such a system are inherently push-based, since you’re listening to incoming Kafka streams, or something similar.
“In a streaming setting, if you want your end consumer to actually be able to interact with the system in a pull-based way (say, by running queries against it when it needs to), you need to introduce some kind of materialization layer where you build an index out of the results.
“Conversely, in a system that doesn’t expose some kind of streaming/tailing mechanism, if you want to know when some data has changed, your only option will be to poll it periodically.”
“Similarly, LIMIT operators can be problematic in the push model. Short of introducing bidirectional communication, or fusing the LIMIT to the underlying operator (which is not always possible), the producing operators cannot know they can stop doing work once their consumer has been satisfied. In a pull system this is not a problem, since the consumer can just stop asking for more results when it doesn’t need any more.”
How Materialize and other databases optimize SQL subqueries by Jamie Brandon (Scattered Thoughts)
“Distinct/Group. Whan a subquery occurs in the from clause and contains a distinct or group by, we have to remove duplicate rows. But there might also be duplicate rows in the outer query that we should not remove. It’s impossible to tell if a duplicate row came from the outer query or the subquery once the two have been joined together.”
“Max1. When a subquery occurs in the select clause, it must return at most one row, otherwise the query is aborted with an error. If we decorrelate the subquery then there is no place in the resulting plan where we can insert the Max1 operator to check the results − if we find two rows with the same variables from the outer it might be because the subquery produced multiple rows for a single outer row or because the outer query already had multiple copies of that outer row.
“(This logic is necessary to deal with cases where there are multiple relations appearing in a scalar expression. There are three reasonable ways that this could have been specced − allow at most one row in each relation, take the product of the relations, or order the relations and pair them up row-wise. Never one for consistency, SQL chose all three options − in select-subqueries, lateral joins and table-valued functions respectively.)”
“In the near term, I think most of these problems can be solved by moving decorrelation into the optimizer rather than having it as a separate pass before optimization. This would allow other optimizations to happen while the plan is still a tree, and would also allow adding many additional decorrelation rules for cases which have simpler solutions.
“But in the long term, I think it’s also worth figuring out how to do plan optimization on graphs. Aside from decorrelation, it also comes up when using CTEs or chains of views. Most databases handles this by making CTEs and views optimization fences, meaning that each is optimized individually but eg filters can’t be pushed down into views. This really limits their usefulness. I’m not aware of much existing work on this problem and most of what I have seen is in the context of datalog which has far fewer tricky corners.”