doesn't do so well at the inverse task, i.e., generating with the probabilities of any token containing a vowel letter OTHER than 'E' zeroed out
#Github #Copilot gives an idea why #Microsoft paid so much for Github. They were after data: Tons of food for their AI, millions of contributors that now 'work' for MS for free.
You publish your code under GPLv3, even AGPLv3? So what? The AI learns from your code and uses it to generate code that is possibly proprietary. Does #GPL forbid this practice? (I don't think so)
That's the M$ way to break copyright law.
It's time for alternatives like @codeberg .
love this sorta disgusting visualization of a self-organizing map https://www.complexity-explorables.org/explorables/yo-kohonen/
Lately I've been reading a lot of children's picture books, over and over? I thought "Goodnight Moon" was pretty spooky, but I had trouble finding anyone writing about that online. @redoak jokingly suggested that I become the conspiracy theorist blogger I want to see in the world, so... I did it. Here's a totally serious take on why "Goodnight Moon" is an esoteric text, from me, a serious scholar of esotericism (aka podcast listener): https://pseudony.ms/blags/goodnight-nobody.html
logit biasing, markov chain style. here I'm doing it with phonetics—basically I check the possible outcomes for each context, and then artificially boost the probability of predictions that have certain phonetic characteristics. (in this case, more /k/ and /b/ sounds)
(tomorrow I'm going to see if stealing alternatives from similar ngrams helps... but I am beginning to more viscerally understand why the solution to language modeling that really caught on is just... More Training Data)
I like having this extra setting to fiddle with! but based on my limited testing, the temperature doesn't really matter once the length of the ngram hits a certain limit, since most ngrams only have one or two possible continuations. like... with word 3-grams, it's pretty difficult to distinguish 0.35 from 2.5
generating with a markov chain using softmax sampling w/temperature (a la neural networks). this is an order 3 character model, and you can really see the difference between low temperature (instantly starts repeating itself) and high temperature (draws from wacky corners of the distribution) (if you've generated text with a markov chain before, it's probably using what amounts to a temperature of 1.0)
The new issue of Bad Quarto's literary magazine is out! Taper #6 offers 26 computational poems, none larger than 2KB, from 23 authors
Taper #6 is thanks to Kyle Booten, Angela Chang, Leonardo Flores, Judy Heflin, and Milton Läufer. This editorial collective determined the theme, selected poems, worked with authors, and did other editorial and production work
All poems are free software
here it is working on an oov ngram ("you ate books" is not an ngram that appears in Frankenstein. all of this is trained on Frankenstein, I guess I forgot to mention that)
another way to find similar ngram contexts: each context has an embedding derived from the sum of positional encoding (they're not just for transformers!) multiplied by "word vectors" (actually just truncated SVD of the transpose of the context matrix). then load 'em up in a nearest neighbor index
(this is cool because I can use it even on ngrams that *don't* occur in the source text, though all of the words themselves need to be in the vocabulary)
Wow, this is a cool little experiment that maps a word-vector space to a text adventure space that you walk around in.
Anyway, large language models (LLMs, like GPT-3) are one of the actual new technologies that technology corporations are racing to get out to market so fast that they've had to sideline and censor all the pesky ethicists and scientists who keep getting in the way by pointing out the litany of actual harms caused by LLMs (discrimination and segregation, wide scale disinformation, environmental impacts of excess computation).
The upsides of LLMs to surveillance capitalism are too high to let social good get in the way of their inevitable production.
In Strange Horizons, Kelly Jennings calls Situation Normal "a hilarious, deeply moving, fast-paced yarn that catches hold of its reader and never lets go."
Poet, programmer, game designer, computational creativity researcher. Assistant Arts Professor at NYU ITP. she/her
Hometown is adapted from Mastodon, a decentralized social network with no ads, no corporate surveillance, and ethical design.