Well this is awkward. Seems that shortly after I posted this here both Mozilla and Google rejected the extension from their stores. Mozilla's primary concern was that the analytics were opt-out, not opt-in, so I'm patching that in and resubmitting.
Typically, Google just said that it "did not comply with their policies" with no additional information, but I'm guessing their concerns are along the same lines.
Meanwhile, if anyone is interested in trying it out, the README has instructions on running the extension locally in Firefox or Chrome.
I built this as an exercise in product management - even though it is just a little extension, I want to run it as if it were a revenue-generating product so that I can gain experience in that area.
Part of that is looking at usage data to identify what parts are working, what needs improvement, retention statistics, etc. I made them opt-out because I would be pissed if someone made a cool thing that I wanted to use but insisted on tracking me with it :)
Because DNT was the quickest solution. I'm actually rewriting that part over the next couple of days to be opt in instead (via a popup the first time the sidebar is opened, I think).
Person posts cool tool they wrote, that they find useful. Shows it to the community, this community, which many may really appreciate. Then gets ripped for not thinking through the complex privacy issues with said tool. Not everything is malicious.
It would be better to give constructive criticism, like:
This is a really cool tool! I might even choose to use it, but I have a few concerns: 1) it sends too much data unfiltered to the algolia search servers. Could you instead make it a button that only then triggers the request or opens another browser window? 2) the analytics are also a concern. In general, you should always make these opt-in; better, ask the user through a dialogue to enable them; even better, request feedback through some other system (github for example).
Let’s try and be nice here and give people the benefit of the doubt.
FWIW, I am taking the privacy and security concerns raised in this thread seriously. In fact, this type of candid feedback is one of the best things about HN. I anticipate it will take me a few days to choose and implement some of the fixes suggested here.
Sharing people's browsing data with third-parties can threaten their lives in some circumstances. There's no excuse for a developer creating browser extensions not to take this threat seriously in 2018. It's negligent.
I did read your comment (and I didn’t downvote you b/c I agreed with you). The reason I wrote this response was because your comment read as being very aggressive in response to an honest attempt at explaining an issue the dev was having. What your comment lacked was any additional information about better approaches.
I get the concern and share it, but your comment only said that what they did was wrong. It didn’t offer any ideas of how to fix it. Telling someone they’re on the wrong road but not telling them how to get to the correct one leaves them just as lost.
Anyone who takes security/privacy seriously shouldn't install any Chrome extension under any circumstances. Many extensions secretly do terrible things that take months or years for anyone to uncover, or the extension is simply sold to a new owner who slips in a malicious payload which is automatically distributed to all users of the extension.
I find it strange how everyone used to be concerned about XSS exploits (including Google themselves who painstakingly implemented reflective XSS prevention, nonce support, and support for XSS-blocking HTTP headers in Chrome) and now they all love XSS as long as it has decent marketing and a clean presentation as an "extension".
I'm still working on making the changes they requested - once I've finished that and submitted a new version, I anticipate it will be a couple days at most until they make the new version available.
One suggestion that came to my mind: As most pages I visit have no comments, I closed the sidebar again. But that way I don't see the comments when there are some. How about adding a setting to let the sidebar open/close automatically if there are comments for a page?
The extension icon should have a number on it that corresponds to the number of comments. My thinking was that you could use that to see if a page had comments or not.
The current implementation will look up any URL you visit so that it can populate the "number of comments" text that appears over the button. The data only goes to 1 server - hn.algolia.com, a HN search API provided by the company Algolia.
Although I trust Algolia not to abuse their service, I understand why you do not. Maybe the "number of comments" display could be opt-in as well? Unfortunately, I don't think there's a way to get that particular UI feature without sending all URLs to some external server.
You can at least start by stripping all query parameters and anchors off of the URL. And instead of automatically piping everything to Algolia, just give the user a damn button.
Having to click a button to see if there are any comments completely changes the usefulness of the addon, doesn't it? zevebs hash approach is the better solution.
Algolia doesn't even have to (intentionally) abuse their service. Request logs can be leaked through software bugs, malicious actors can gain access to their servers, etc.
Trying to find matches for URL hashes would be a much nicer approach, but you would need a database of hashed URLs for that, so it's not a trivial fix.
> Unfortunately, I don't think there's a way to get that particular UI feature without sending all URLs to some external server.
Look up the protocol that Google's SafeBrowsing[0] project uses. The short version is that your extension should check visited URLs against a locally-downloaded database of HN-posted URLs. For SafeBrowsing there's a bit more optimisation (e.g. hashing URLs & storing only hash prefixes to reduce the size of the database), but that's the short version.
At least change it so it sends a hash of the URL to a server that knows the hashes of all the HN submitted posts. That way if there is no match you'd have to have hashes of all legal URLs to leak anything.
Suppose I'm an evil overlord and someone in my organization has been tipping off the FBI about my upcoming operations. I'm pretty sure it is one of my work-at-home minions.
If I can obtain the hashes of the URLs that my minions have visited, I can look for a minion that has in their history the hash of https://tips.fbi.gov/ and now I've got a good suspect.
You'd have to have a lot more than just that hash, the log would at least have to include a static IP or something that you can isolate by window-of-opportunity, for instance all the other minions were at a ballgame and the timestamp indicates that that one minion that wasn't at the ballgame visited tips.fbi.gov right then from an IP not associated with a stadium hotspot.
Regardless, if you're going to leak stuff on 'evil overlord's organization you'd better make sure you don't do it from anything that can be associated with you, so not your laptop, not your IP, not your browser and certainly not with all kinds of weird plugins installed. Burner device and a location and time chosen so it could be anybody in 'the organization' leaking.
The first submission on the "New Links" page that is at least 1 day old is number 1140 currently. The first item is 4137 days old. That yields an estimate of less than ≈4.7 million submissions so far, many of which are likely to be duplicates, flagged or devoid of comments. The hashes of all relevant posts likely fit in less than a megabyte, no need to contact a server, except to add fresh posts to the list.
But that way they would never have comments at all..., you'd have to at least store them as long as comments could be entered, so until the thread enters archival state.
It could do that - I built it to satisfy my own use case, which was that I wanted to see the HN comments side-by-side with the content while I was browsing.
Why send each URL to a server to be checked, instead of doing a periodic download of the (very small) list of HN links and comment counts, to be checked offline like an ad blocker? It's probably 5kb compressed.
What makes you think it's a very small list? HN has been around since 2007, with a lot of submissions. Any guesses on the size of that? I think it's bigger than
You can sort HN by date, and few URLs are updated every day. So you can push a new bloom filter every day and a different list of updates every 5m. Then just check your URL against both of them.
Well... including comments it looks like we are pushing 16.3 million posts. The id in the url is sequential. If you are saving url, HN id, and comment count, that's probably no more than a couple megs, if even that.
~391 MB if we store SHA-1 hashes of the URLs (160 bits each) and HN ids and assume 16.3 million posts[1]. (Probably less, since, as you said, some posts are just comments.) If we're okay submitting one out of every hundred URLs as a SHA-1 hashed value to an external server, we can reduce that further to ~18 MB with a bloom filter[2].
I don't think we need either a strong hash or to submit hashed URLs to a server. We can use an ordinary hash and the only URLs we need to submit to a server are story requests to HN, if we do this thing like this:
Include in the extension a hash table construction as follows:
foreach ID of an HN story submission
URL = the URL of the submitted story
URL = normalize(URL)
insert_into_hash_table(URL, ID)
insert_into_hash_table(key, val) is a function that inserts val into a hash table with key key. The hashing function does not need to be cryptographically secure.
normalize(URL) is a function that takes a URL and normalizes it. What normalize means in this context is a little fuzzy, but the basic idea is that if URL_1 and URL_2 are different URLs to the same article, normalize(URL_1) == normalize(URL_2).
NOTE: what you include with the extension is the hash table itself. Conceptually it is probably just a sparse array containing HN IDs, with maybe a little more depending on how collisions are handled.
In the extension, do this:
URL = normalize(URL_of_current_page)
ID_list = lookup_URL_in_hash(URL)
foreach ID in ID_list
story = get_HN_story(ID)
if (normalize(URL_of_story(story)) == URL
show_story_comments(story)
The hash is only used for data retrieval from a local hash table, so does not need to be cryptographically secure.
After the hash lookup we have a list of candidate stories on HN that might match the browser story. It's a list because due to collisions there might be more than one HN story with matching hash.
Note that all that is ever fetched from the server during operation of this are HN stories, so there is minimal information leakage.
Don't hash tables typically store the key itself in the table though? Wouldn't that take up _more_ space per entry than a 160-bit hash? Using a sufficiently collision-resistant hash function allows you to eliminate the need to store URLs entirely, which in theory should reduce the size of the hash table significantly.
Whether or not you need to store keys depends on how you handle collisions.
For instance, if we had a hash table whose keys were names, and whose values were telephone numbers, we'd probably have to store keys with the phone numbers so that in the case of a collision we could figure out which phone number matches the search key.
If, on the other hand, we had a hash table that stored record numbers of employee records from our employee database, keyed by employee name, then we probably would not need to store keys with the hash table values. If there is a collision, we can just retrieve all of the colliding records from the database. Those records will contain the employee name, and we can use that to figure out which is the right one.
For the HN comment extension we are closer to the second case. The HN story contains the URL, so in the case of a collision we can fetch all the colliding HN stories and see which one is the right one.
But wasn't the 16 million estimate based on the increasing ID numbers, which includes comments and self-posts, neither of which don't need to be hashed or checked? The number of submitted URL's that need to be checked against is presumably much smaller, and the number of those that have at least one comment is much smaller than that.
Edit: Estimating quickly from the new submissions for the last couple hours, it looks like there are on the order of 100 submissions per hour. Of these, it seems like less than 10% have any comments. So if we assume that HN has been up for 12 years (100,000 hours) we get about 1 million commented URL's with comments, and that's probably an overestimate (dups, increasing usage, etc).
No. In fact, the development cycle on this was pretty awful - due to browser extensions' strict CSP, I couldn't even compile the CLJS with {:optimizations :none}, so every code change took ~20 seconds to recompile.
Do you know any good resources on REPL-driven ClojureScript development? Preferably with a REPL that lives in the page environment so that browser variables etc. can be accessed.
Interesting, looks like they got :optimizations :none to work for background scripts but not content scripts (which makes sense). Figwheel support sounds great, I'll look into adding that to Looped In.
As other comments here point out, there are significant privacy issues with sending every visited URL to a "trusted" server to check for comments. There are also load issues for the server that is getting all the unwanted requests. So if doing this, you'd probably want at least the initial lookup to be local.
So, can this be done with an initial download of some small number of megabytes, then incremental updates of a few kilobytes as often as desired? I think so.
Guessing at numbers (see other comments here), there are probably fewer than 1,000,000 URL's that have comments associated with them on HN. For each, you store hash of the canonicalized URL and a comment count. Collisions aren't deadly, so you could probably get down to 8B per hash (7B for hash, 1B for count). Updates can be a list of all new and revised URL-hash-counts since a given date.
Lookup the hash of the URL of each visited page with binary search on the 7B prefix in the ~8MB ordered list of data. If found, report the number of comments in something clickable that loads a sidebar. The only data that leaves the machine is based on the active click to load the comments. Maybe store a "false positive list" so that the rare collisions are only visible to the user once. Maybe use a bit for "visited" so you can distinguish pages with new comments?
What does your development workflow for CLJS browser extensions look like? Do you have to rebuild and reload the extension on your development machine each time you change the code? Is it possible to use Figwheel when writing browser extensions?
I'm still working out the kinks in the workflow. I believe it is possible to use Figwheel while writing browser extensions [0], but I haven't set that up. I have `lein cljsbuild auto` running in the background to automatically recompile the JS when I change the source code, and use Mozilla's web-ext utility [1] to automatically reload the web extension once the JS has compiled.
This system has some disadvantages, though. The main one is the glacial feedback loop - due to strict CSP restrictions in web extensions, I had trouble compiling the CLJS with {:optimizations :none}, so each save-compile-reload cycle takes ~30 seconds from when I save the source file to when I see the results of the change in the browser. I also did not figure out how to set up a REPL environment connected to the code running in the web extension.
76 comments
[ 5.7 ms ] story [ 138 ms ] threadTypically, Google just said that it "did not comply with their policies" with no additional information, but I'm guessing their concerns are along the same lines.
Meanwhile, if anyone is interested in trying it out, the README has instructions on running the extension locally in Firefox or Chrome.
Part of that is looking at usage data to identify what parts are working, what needs improvement, retention statistics, etc. I made them opt-out because I would be pissed if someone made a cool thing that I wanted to use but insisted on tracking me with it :)
It would be better to give constructive criticism, like:
This is a really cool tool! I might even choose to use it, but I have a few concerns: 1) it sends too much data unfiltered to the algolia search servers. Could you instead make it a button that only then triggers the request or opens another browser window? 2) the analytics are also a concern. In general, you should always make these opt-in; better, ask the user through a dialogue to enable them; even better, request feedback through some other system (github for example).
Let’s try and be nice here and give people the benefit of the doubt.
FWIW, I am taking the privacy and security concerns raised in this thread seriously. In fact, this type of candid feedback is one of the best things about HN. I anticipate it will take me a few days to choose and implement some of the fixes suggested here.
Sharing people's browsing data with third-parties can threaten their lives in some circumstances. There's no excuse for a developer creating browser extensions not to take this threat seriously in 2018. It's negligent.
I get the concern and share it, but your comment only said that what they did was wrong. It didn’t offer any ideas of how to fix it. Telling someone they’re on the wrong road but not telling them how to get to the correct one leaves them just as lost.
And let's all try not to get the users of our software imprisoned or killed: https://boingboing.net/2018/01/28/30000-accused.html
I find it strange how everyone used to be concerned about XSS exploits (including Google themselves who painstakingly implemented reflective XSS prevention, nonce support, and support for XSS-blocking HTTP headers in Chrome) and now they all love XSS as long as it has decent marketing and a clean presentation as an "extension".
One suggestion that came to my mind: As most pages I visit have no comments, I closed the sidebar again. But that way I don't see the comments when there are some. How about adding a setting to let the sidebar open/close automatically if there are comments for a page?
I'd rather not have a browser extension that sends everything I visit to some US servers - but I do like the idea behind this.
There's so much that can go wrong with that, especially if this stuff eventually shows up in some data leak.
Here's some examples:
1. URLs can contain sensitive data, which will tie your session to your person (facebook URLs etc)
2. This could create vulnerabilities in (for example) checkout flows that pass data via URLs
3. I'd rather not make the job of intelligence services easier than it already is
4. Sometimes I clear my browser history for a reason
Trying to find matches for URL hashes would be a much nicer approach, but you would need a database of hashed URLs for that, so it's not a trivial fix.
Look up the protocol that Google's SafeBrowsing[0] project uses. The short version is that your extension should check visited URLs against a locally-downloaded database of HN-posted URLs. For SafeBrowsing there's a bit more optimisation (e.g. hashing URLs & storing only hash prefixes to reduce the size of the database), but that's the short version.
This secures the end user's privacy.
[0] https://safebrowsing.google.com/
Suppose I'm an evil overlord and someone in my organization has been tipping off the FBI about my upcoming operations. I'm pretty sure it is one of my work-at-home minions.
If I can obtain the hashes of the URLs that my minions have visited, I can look for a minion that has in their history the hash of https://tips.fbi.gov/ and now I've got a good suspect.
You'd have to have a lot more than just that hash, the log would at least have to include a static IP or something that you can isolate by window-of-opportunity, for instance all the other minions were at a ballgame and the timestamp indicates that that one minion that wasn't at the ballgame visited tips.fbi.gov right then from an IP not associated with a stadium hotspot.
Regardless, if you're going to leak stuff on 'evil overlord's organization you'd better make sure you don't do it from anything that can be associated with you, so not your laptop, not your IP, not your browser and certainly not with all kinds of weird plugins installed. Burner device and a location and time chosen so it could be anybody in 'the organization' leaking.
A reasonable hash is going to be 64 bytes or more to avoid collisions, the total DB would be a least a couple of 10's of megs.
[1]: https://www.google.com/search?q=(160+bits+%2B+32+bits)+*+16....
[2]: https://hur.st/bloomfilter?n=16300000&p=0.01
Include in the extension a hash table construction as follows:
insert_into_hash_table(key, val) is a function that inserts val into a hash table with key key. The hashing function does not need to be cryptographically secure.normalize(URL) is a function that takes a URL and normalizes it. What normalize means in this context is a little fuzzy, but the basic idea is that if URL_1 and URL_2 are different URLs to the same article, normalize(URL_1) == normalize(URL_2).
NOTE: what you include with the extension is the hash table itself. Conceptually it is probably just a sparse array containing HN IDs, with maybe a little more depending on how collisions are handled.
In the extension, do this:
The hash is only used for data retrieval from a local hash table, so does not need to be cryptographically secure.After the hash lookup we have a list of candidate stories on HN that might match the browser story. It's a list because due to collisions there might be more than one HN story with matching hash.
Note that all that is ever fetched from the server during operation of this are HN stories, so there is minimal information leakage.
For instance, if we had a hash table whose keys were names, and whose values were telephone numbers, we'd probably have to store keys with the phone numbers so that in the case of a collision we could figure out which phone number matches the search key.
If, on the other hand, we had a hash table that stored record numbers of employee records from our employee database, keyed by employee name, then we probably would not need to store keys with the hash table values. If there is a collision, we can just retrieve all of the colliding records from the database. Those records will contain the employee name, and we can use that to figure out which is the right one.
For the HN comment extension we are closer to the second case. The HN story contains the URL, so in the case of a collision we can fetch all the colliding HN stories and see which one is the right one.
Edit: Estimating quickly from the new submissions for the last couple hours, it looks like there are on the order of 100 submissions per hour. Of these, it seems like less than 10% have any comments. So if we assume that HN has been up for 12 years (100,000 hours) we get about 1 million commented URL's with comments, and that's probably an overestimate (dups, increasing usage, etc).
And there is also: https://github.com/powerpak/hn-sidebar which was in the Chrome Store but not anymore...
Any idea why the earlier extension was shut down by YC?
I tried to use clojurescript for Chrome extensions few years ago but failed to configure fast reload cycle (either repl or figwheel).
Do you know any good resources on REPL-driven ClojureScript development? Preferably with a REPL that lives in the page environment so that browser variables etc. can be accessed.
https://chrome.google.com/webstore/detail/kiwi-conversations...
Also "researches" Reddit, Product Hunt and Google News. And only on demand
So, can this be done with an initial download of some small number of megabytes, then incremental updates of a few kilobytes as often as desired? I think so.
Guessing at numbers (see other comments here), there are probably fewer than 1,000,000 URL's that have comments associated with them on HN. For each, you store hash of the canonicalized URL and a comment count. Collisions aren't deadly, so you could probably get down to 8B per hash (7B for hash, 1B for count). Updates can be a list of all new and revised URL-hash-counts since a given date.
Lookup the hash of the URL of each visited page with binary search on the 7B prefix in the ~8MB ordered list of data. If found, report the number of comments in something clickable that loads a sidebar. The only data that leaves the machine is based on the active click to load the comments. Maybe store a "false positive list" so that the rare collisions are only visible to the user once. Maybe use a bit for "visited" so you can distinguish pages with new comments?
The numbers seem surprisingly manageable.
I've opened an issue to implement this here: https://github.com/jdormit/looped-in/issues/4
As time allows, I will implement this later this week.
What does your development workflow for CLJS browser extensions look like? Do you have to rebuild and reload the extension on your development machine each time you change the code? Is it possible to use Figwheel when writing browser extensions?
This system has some disadvantages, though. The main one is the glacial feedback loop - due to strict CSP restrictions in web extensions, I had trouble compiling the CLJS with {:optimizations :none}, so each save-compile-reload cycle takes ~30 seconds from when I save the source file to when I see the results of the change in the browser. I also did not figure out how to set up a REPL environment connected to the code running in the web extension.
Lots of room for improvement here, basically.
[0]: https://github.com/binaryage/chromex-sample#chromex-sample-p...
[1]: https://developer.mozilla.org/en-US/Add-ons/WebExtensions/Ge...