* It does a remarkably good job of context switching between programming languages based on the semantics of the question! If the question is about SQL it often includes SQL in < code > tags. If it’s about JavaScript it will include JavaScript! The syntax isn’t perfect due to the tokenizer mangling some things but it’s pretty close!
* The English grammar isn’t perfect but it’s pretty good.
* It doesn’t seem to lose closing track of closing tags and quotes.
* It's learned to sometimes pre-emptively thank people for their answers and to "edit" in "updates" at the end of the post.
If you find any interesting ones you can share them with the permalink! Use the "Fresh Question" button to load a new one.
> Answering Questions Right now the model only generates questions. In version 2 I want to train it to answer questions. If I could get this working it'd actually become a useful tool instead of a fun toy.
Looking forward to that part :D
I mean, those answers are probably not going to be correct, but I wonder how close they will be to something useful.
Yes, many times the questioner does not actually need an answer to the question, he just needs to look a little closer to the situation, which is potentially able to be automated. But one should not disguise such automation as an 'answer': more like a query autocheck but more tooled-up.
I wonder what percentage of questions just need a correctly working example because the questioner is unsure of how to use a given API. Automation of this I imagine could actually be doable.
Thanks, brilliant work! Some questions are downright hilarious (see a suite of automated packaging techniques [0]), and the broken English just adds extra ESL credibility to the questions.
[0]
>I want to write an update statement with a sequence of values I can run through a database. i've written the below code to broken up my character string into the columns. All of the articles i've read seem to suggest that I 'll need a suite of automated packaging techniques for my environment to all update the database.
> Originally, I wanted to predict the number of upvotes and views questions would get (which, intuitively, I thought would be a good proxy for their quality). Unfortunately, after working on this for about a week straight I've come to the conclusion that there is no correlation between question content and upvotes/views.
> I tried several different models (including adapting an AWD_LSTM classifier, a random forest on a bag of words, and using Google's AutoML) and none of them produced anything better than random noise.
> I also tried using myself as a "human classifier" and given two random questions from StackOverflow I can't predict which one will be more popular.
It’s a different model, the AWD_LSTM. The inventor of this model spoke about GPT-2 on this podcast and talked a bit about the differences: https://overcast.fm/+Goog4jsR8
You are a very well trained neural net... The concept is based off of actual Neurons in our brain. Can't tell if you're serious or trolling though lol.
Reminds me of those online chatbots I used to torture back 10 or 15 years ago. One I started asking about personal information about its creator. It was remarkably evasive, constantly attempting to switch the subject.
Fascinating. I wonder if our current discussion boards on the interwebs can survive the coming influx of content like this and the next generations of it that follow.
There are a lot of SO questions posted by very weak non-native speakers of English and some of these are hard to distinguish from those. Kind of scary!
What possible positive outcomes do you see for this kind of (admittedly inevitable) capability?
I was clicking through a couple funny ones but as soon as I got one[1] that fell into this uncanny valley I immediately forgot this was generated and tried to understand it, getting super confused.
I am actually a bit worried that I’m already starting to see search engine traffic coming in...
I hope that the good will outweigh the bad. I’d love to create an answer generator, for example.
Once enough questions are generated I’m going to try creating a classifier to see if a neural net can differentiate between real questions and fake ones.
> I am actually a bit worried that I’m already starting to see search engine traffic coming in...
We can discuss hypothetical systems that could maliciously flood us with generated content. The creator of this particular service which is being discussed here and now could also begin taking steps to ensure that his creation does not inadvertently create a problem for some hapless Google user.
Well, no. You have to read further up the thread to see the issue I was referring to.
>I wonder if our current discussion boards on the interwebs can survive the coming influx of content like this and the next generations of it that follow.
Yes the robots.txt is a good and trivial step he could take to ensure well behaved robots do not pick up his content. So your comment suggesting robots.txt is a good comment in its narrow frame, but one that missed the larger picture. That minor problem is solved. The interesting problem is of a different nature.
Every good invention can be terrifying if it falls in the hands of bad guys (Nuclear technology for example).
It's true for AI also. I am sure bad guys must be training similar AI agents by only feeding fake news, conspiracy theories etc. and it's easy to build AI agents as there is so much Open Source material online about AI.
Think things like election meddling. Propagating truly fake news to cater to the emotions of what people simply want to be true. Humans are weak against Confirmation Bias, ten minutes on Facebook will show you for sure.
There is immense value in training these to synthesize test data sets for sensitive information you can't safely put in a preprod environment.
Health information would be the main case I can think of now.
Having synthesized data for testing new services in govt would be a huge improvement.
De-identification is basically impossible and there are a bunch of companies who will lie to you if you pay them to, but synthesized data covers many use cases for de-identification and for homomorphic encryption.
Unfortunately I’ve come to the conclusion that upvotes on Stackoverflow aren’t correlated with question content (or I’m not skilled enough to be able to differentiate between “good” and “bad” questions). Check out the linked write up for more detailed info.
I think the original comment is being sarcastic and suggesting that the actual humans that close discussions and mark them as "off topic" don't understand the question and perform these actions at random. This is a sentiment shared by many who don't "live" in those types of forums.
Could you not also train a classifier that correlates question content with mod decision? Questions like "what is the best X? " that are obviously subjective
, for example.
Maybe even some kind of crazy generative model that learns to post questions that aren't closed by the AI moderator!
The way the language model is trained is by rewarding it for correctly predicting the next word in a sequence.
The output of the model is a predicted probability distribution of the next word and a “state” — the next iteration takes the state output of the previous interation and generates another word and state (and this process repeats many times).
Since there’s a probabilistic dimension, what may have happened in this case is that it happened to repeat once by chance and the model had learned that if something repeats 2x it’s likely that it will repeat a third, fourth, and fifth time.
Basically it’s just trying to game the loss function which rewarded it for predicting the next word in the sequence correctly.
Thanks for the explanation. Your description superficially reminds me of a Markov chain (https://en.wikipedia.org/wiki/Markov_chain). Is this related or is it totally different?
I haven't read the paper the work is based on, but if the RNN outputs a probability distribution for the next letter/word then they form Markov Chains (since then they only depend on the current state and not the previous state)!
RNNs are just fancy parametric functions that take a (state, input)-pair and return a new (state', output)-pair.
I think it would still be considered a "Does Not Exist"-valid website if the generated questions would have some auto-formatter for the code. Main issue I see is extra spaces everywhere, often in a syntax breaking way (and missing spaces for formatting) (not that all SO questions have those).
Yeah this is a shortcoming of the tokenizer. It splits things up in ways that are not 1:1 mappable back to their source unfortunately.
I did a bit of post-processing to get it formatted a bit better (re-combining the “would“ and “n’t” tokens and changing html tags to markdown for example) but there’s still room for improvement.
Spacing specifically is different based on the context. Outside of code blocks you want a space after a period. Inside you probably don’t. But since the tokenizer has one in both places there’s no opportunity for the neural net to learn this (it can’t see any difference). And my naive formatted doesn’t know the difference either. (If you’re curious you can find it in the JS file)
I updated my regexes to clean up some of the tokenizer noise last night. So many of the formatting in the code snippets should look a bit more natural now.
Oh, great, now I know how my clueless questions look like to a knowledgeable person! Example:
>"I need to create an image from a imported wav file (for a user - friendly format find enough header for the cookie). I looked for a solution, but that didn't work either."
I protest getting -4 points. They used github and stackoverflow. Wrote a function to connect the two based on tags and then randomly generate a question off of that. It's lame. Do something useful or cool.
142 comments
[ 3.0 ms ] story [ 200 ms ] threadFull writeup available here: https://stackroboflow.com/about/index.html
Interesting things I’ve noticed so far:
* It does a remarkably good job of context switching between programming languages based on the semantics of the question! If the question is about SQL it often includes SQL in < code > tags. If it’s about JavaScript it will include JavaScript! The syntax isn’t perfect due to the tokenizer mangling some things but it’s pretty close!
* The English grammar isn’t perfect but it’s pretty good.
* It doesn’t seem to lose closing track of closing tags and quotes.
* It's learned to sometimes pre-emptively thank people for their answers and to "edit" in "updates" at the end of the post.
If you find any interesting ones you can share them with the permalink! Use the "Fresh Question" button to load a new one.
Looking forward to that part :D
I mean, those answers are probably not going to be correct, but I wonder how close they will be to something useful.
[0] >I want to write an update statement with a sequence of values I can run through a database. i've written the below code to broken up my character string into the columns. All of the articles i've read seem to suggest that I 'll need a suite of automated packaging techniques for my environment to all update the database.
What s the best way to update the column ids?
Thanks
> Originally, I wanted to predict the number of upvotes and views questions would get (which, intuitively, I thought would be a good proxy for their quality). Unfortunately, after working on this for about a week straight I've come to the conclusion that there is no correlation between question content and upvotes/views.
> I tried several different models (including adapting an AWD_LSTM classifier, a random forest on a bag of words, and using Google's AutoML) and none of them produced anything better than random noise.
> I also tried using myself as a "human classifier" and given two random questions from StackOverflow I can't predict which one will be more popular.
I’ll have to look and see if there’s an archive of them available.
Reminds me of those online chatbots I used to torture back 10 or 15 years ago. One I started asking about personal information about its creator. It was remarkably evasive, constantly attempting to switch the subject.
There are a lot of SO questions posted by very weak non-native speakers of English and some of these are hard to distinguish from those. Kind of scary!
What possible positive outcomes do you see for this kind of (admittedly inevitable) capability?
[1] https://stackroboflow.com/#!/question/13535
I hope that the good will outweigh the bad. I’d love to create an answer generator, for example.
Once enough questions are generated I’m going to try creating a classifier to see if a neural net can differentiate between real questions and fake ones.
> I am actually a bit worried that I’m already starting to see search engine traffic coming in...
We can discuss hypothetical systems that could maliciously flood us with generated content. The creator of this particular service which is being discussed here and now could also begin taking steps to ensure that his creation does not inadvertently create a problem for some hapless Google user.
>I wonder if our current discussion boards on the interwebs can survive the coming influx of content like this and the next generations of it that follow.
Yes the robots.txt is a good and trivial step he could take to ensure well behaved robots do not pick up his content. So your comment suggesting robots.txt is a good comment in its narrow frame, but one that missed the larger picture. That minor problem is solved. The interesting problem is of a different nature.
Brilliant, now you only need to come up with a way to use this for good and keep (at least slightly) ahead of the cost in the long run.
At some point this technology will extend into what's left of print, then talk radio, then TV. An endless supply of Markov punditry.
http://approximatelycorrect.com/2019/02/17/openai-trains-lan...
thispersondoesnotexist.com thisresumedoesnotexist.com
I actually hadn't seen thisresumedoesnotexist.com yet; but I loved https://thiscatdoesnotexist.com and https://thisrentaldoesnotexist.com
Oh my god, some of these look terrifying, pure nightmare fuel.
Like the Airbnb, it's StyleGAN+GPT-2 (finetuned in this case on anime plot synopses+summaries: https://www.gwern.net/TWDNE#gpt-2-anime-plot-synopses-for-gp... ).
I'm currently training an improved 'portrait' anime StyleGAN to fix up some of the faces' issues.
Health information would be the main case I can think of now.
Having synthesized data for testing new services in govt would be a huge improvement.
De-identification is basically impossible and there are a bunch of companies who will lie to you if you pay them to, but synthesized data covers many use cases for de-identification and for homomorphic encryption.
https://stackroboflow.com/#!/question/973
https://stackroboflow.com/#!/question/29584
``` Cat cat = new Cat(); Cat cat = new 2nd Cat ");" ```
Unfortunately I’ve come to the conclusion that upvotes on Stackoverflow aren’t correlated with question content (or I’m not skilled enough to be able to differentiate between “good” and “bad” questions). Check out the linked write up for more detailed info.
I think the original comment is being sarcastic and suggesting that the actual humans that close discussions and mark them as "off topic" don't understand the question and perform these actions at random. This is a sentiment shared by many who don't "live" in those types of forums.
Maybe even some kind of crazy generative model that learns to post questions that aren't closed by the AI moderator!
Sidenote, would this technically count as a neural network with only one weight (which is randomly initialized)?
Is there a reason why? (I don't know anything about AI.)
The output of the model is a predicted probability distribution of the next word and a “state” — the next iteration takes the state output of the previous interation and generates another word and state (and this process repeats many times).
Since there’s a probabilistic dimension, what may have happened in this case is that it happened to repeat once by chance and the model had learned that if something repeats 2x it’s likely that it will repeat a third, fourth, and fifth time.
Basically it’s just trying to game the loss function which rewarded it for predicting the next word in the sequence correctly.
RNNs are just fancy parametric functions that take a (state, input)-pair and return a new (state', output)-pair.
I did a bit of post-processing to get it formatted a bit better (re-combining the “would“ and “n’t” tokens and changing html tags to markdown for example) but there’s still room for improvement.
Spacing specifically is different based on the context. Outside of code blocks you want a space after a period. Inside you probably don’t. But since the tokenizer has one in both places there’s no opportunity for the neural net to learn this (it can’t see any difference). And my naive formatted doesn’t know the difference either. (If you’re curious you can find it in the JS file)
Without too much sarcasm I have received support requests that were far too close to this.
---
Laughing way to hard, but is it having a stroke? https://stackroboflow.com/#!/question/14791
YMMD :D
(^.^ this “so-SO” comment made me smile.)
>How can I do this software?
Although it sounds more like a question from Quora.
>"I need to create an image from a imported wav file (for a user - friendly format find enough header for the cookie). I looked for a solution, but that didn't work either."
https://stackroboflow.com/#!/question/4781
A common problem for everyone, i'm sure.
Now that is comedy!
Sounds like jumpstarting your userbase would be easy, once you allow users to define other users :)
I can feel the admin is different
You sure you didn't just accidentally create a self-aware AI? Forgot to permalink sadly