Ask HN: What interesting problems are you working on?
This is a follow up of this thread ( https://news.ycombinator.com/item?id=22174828 ) which got some amazing responses but considering it's been almost two years, It's time for new edition
I'd love to hear about what interesting problems (technically or otherwise) you're working on -- and if you're willing to share more, I'm curious how you ended up working on them. This time, with a twist. Please include web technologies too if it's really niche and not a lot of people know about it
633 comments
[ 1.4 ms ] story [ 353 ms ] threadI’m a contractor, I work through my company. It’s way more tax efficient than being perm.
But, it could be even more efficient.
So, I did some research. Turn out, where I am, a business can have multiple business codes. Activities the company is registered as performing. Now, I was and still am a software development company. But I also added artistic production as a business code.
Why artistic productions? Well, because it’s the most liberal business code for expenses.
As a software company, I can expense travel for some things, the most useful for me being conferences. But that means I need to buy the conference ticket even if I’m not interested and can only expense accommodations for the duration of the conference. That’s good but a bit restrictive.
Now, as a artistic production company, I’m not limited by that. As long as I can prove I traveled in order to produce, well, just about anything that can be construed as being art, I can expense travel and I can expense accommodations for as long as I need. You can’t rush art, you know how it is.
So, obviously, first thing I did, I whipped up a blog, posted some random pictures every day and called it good. I’m not using company money to travel the world you see, I’m using company money in order to produce art. Haven’t paid a euro of my own money on accommodation or travel for close to a year now. And the best part, I get to deduce this from my profit.
I love the tax system. I really do.
But, doing this, I got curious. I thought, hey, why not make it better. So I thought let’s see what all the rage is with dalle and stable diffusion and all that. So, what I’m doing now is, I’m changing my approach. My art will no longer be just pictures taken quickly with my iphone. No no. I’m getting the picture and I’m sending it to an API I built. The API runs some computer vision, labelling what it sees, some classification, some stuff with the pic metadata, tries to put it all together and turns it into a description of the picture.
The description of the picture I took is used as a prompt for stable diffusion. And that result ends up being the art. I’d like to thank of my art as a modern commentary on the age old “is art imitating life or is life imitating art?” - At least that’s what I told my accountant and he concerned, it would stand up in the case of an audit!
So, yeah, tldr, I’m working on image to text and text to image to expense my nomad lifestyle on the company.
I take out only 11k or so per year from the company as personal income. That’s the tax free bracket. The rest I use is just expenses. And I pay myself minimum wage as well. Also tax free, but qualifies me for national insurance :D
I love the tax system!
Do you think the tax money serves any societal good at all? Is it worth paying any tax or would you rather avoid all of it if you could?
In the UK there’s something called IR35. Without getting into to much detail, the client will decide if you fall inside or outside. If you are inside or outside you will pay different amounts of tax.
Not that long ago I was offered £1000 per day inside. That seemed like a really good rate to me. Or so I thought. In a month, 21 working days, that’s £21k per month, how much can taxes be right? Let’s call it £17k - I mean c’mon, £4k per month in taxes is already ludicrous. I couldn’t possibly be more. Right?
Please go to the link and plug in £1000 per day. Look at what it says you get cash in hand at the end of the month and tell me, does that seem ok? Is that tax worth paying?
https://www.contractorcalculator.co.uk/insideir35contractorc...
Those numbers don't seem particularly significant either way to me, coming from the US. Our tax brackets cap out at about 40% right now, but they used to be in the high 90%s decades prior -- and that's just the federal, and then some states have a bunch of other tax burdens on top of that. But personally, I'm economically pretty left-leaning and would prefer expanded social services rather than private wealth accumulation, so I doubt I'm representative of what the average person would consider reasonable vs excessive when it comes to taxes.
That's not really the question though. I get that a lot of people (probably most?) wouldn't pay more taxes than they have to, and frankly I can hardly fault them for that. Everything from our genes to our industrialized capitalist societies encourages in-group prioritization and selfish behaviors. That's just how this world that we've made works, and even the most fervent idealists dare not dream of making the entire world pool and share income. We just ain't built that way, and that's just opening the floodgates to insane corruption.
The more interesting question, IMO, is whether ANY of it is worth paying for collaboratively. If I could design my own tax system (and have a billion minions happily working in it...) the upper bracket would be insanely high, like 99% or some such, but there would also be a high degree of choice, per taxpayer, for some portion of their funds. Like there might be a mandatory % going to roads, defense, schools, health -- basic infrastructure, broadband -- but then each taxpayer would get to choose where the rest of it goes (say, space, basic research, the arts, land trusts, energy development, whatever). Two millionnaires would maybe each keep 50% of their income, spend 25% on basic infra, but be able to choose (within constraints) how the remaining 25% is spent. One than decide to split his 25% among various pet causes, the other might give all of hers to increase defense R&D. Basically a democratic Robin Hood, where an armed bandit takes your money and gives it to charity, but asks you, "Which charity?". Heh.
That's just me.
My question to you, as someone who goes out of their way to avoid paying taxes, is... how would you do it? Is some of it still worth it, after all the pork-barrel spending and corruption and inefficiency and bureaucracy? Should it all be private? Like take what you're doing, how would you ideally scale it up to country-level with a few million taxpayers?
That is nothing short of insane to me. That is almost 3 quarters of your money taken by the government. Maybe you can live with that. I can’t. It’s insane to me. I simply refuse to participate in that.
As for taxes, I get your point. Sounds good in theory. Until you realise people like me are not that uncommon. That 25% I still have a say on is going to go to my charity. A social change sort of charity. A charity aimed at offering educational alternatives to promising young leaders. The sort that will grow up to question the justice of your tax regime and advocate for it’s dismantling. And with a bit of luck and push from other rich people, give it a few decades and it will go away.
But to answer you:
1. 1% flat tax, applied to income for people and profit for businesses.
€1000 goes into your business -> you’re left with €990 -> you take them all out as dividends, you’re left with €980.1
This will provide an annual bulk income.
2. Flat transaction fee. Every time money changes hands, the government skims 0.1€ from it.
This will provide a steady income stream.
You buy a laptop, you pay €1000 for it, the vendor only gets €999.9 - This is not something operators worry about, this is done automatically by the banks.
But this won’t be enough to <insert something here>. Well, tough luck. Better start prioritising. Start by cutting foreign aid. Start by cutting aid to ngos shipping migrants from africa, start by cutting welfare to illegals, get rid of the useless government bureaucracy like “period dignity officers”, plenty of stuff to cut.
Make do with less, that’s what I would tell the government.
Are there any examples of a system like you're describing working out in reality? Doesn't have to be a country, but maybe a local government, a private community, a membership club, etc.?
Sounds like a libertarian fantasy out of Atlas Shrugged or Bioshock... but hey, we all have our dreams, and my utopia would be your dystopia, lol.
In the US, at least, companies provide a lot of benefits that contractors don't get and have to buy out-of-pocket... but not sure if that's what the difference comes from in this case.
Sure. The IRS could change things. But you're also able to do a lot more in America that they can't track. Other countries (e.g. Sweden, etc) are far "simpler", and homogeneous. For example, the depth at which one can trade various instruments, produce companies, etc is far far greater in America than almost anywhere else. This necessitates a somewhat (perhaps not as much as today) complicated tax code.
I would say this is very productive. The tax code isn't actually the problem. It's corporate interests like intuit who, through regulatory capture, make it impossible to truly solve the problem. Honestly, it cannot get easier than a 1040EZ which is what mostly everyone uses. In fact the 1040EZ is so easy you basically fill in the things to confirm the number the IRS already has is correct. OP, like myself, need more complicated solutions. I have a fairly vast portfolio of different investment types and OP has a business. In both cases, investing time into making the IRS's life difficult pays a return on par with bonds.
Concerning taxes though, I'll have to (quasi) side with you. 1040EZ is as easy as it gets. One could of course argue that in that case, in which the IRS has the numbers already, why do you have to be forced to do your taxes at all (think Germany).
The greatest fallacy of the pseudo-intelligent is comparing different first world countries to each other without considering demographics. It's a fallacy you have committed, along with everyone else who says "America is backwards lol". That is why it is a trope. It has nothing to do with American exceptionalism and everything to do with a relatively poor understanding of how we arrived here.
America is a punching bag for the rest of the first world because it has problems literally no other first world country has to face. Problems that are too innumerable to list here. Without considering the various reasons America is a Special Case (TM) in many ways, you're missing the greater point. Sure we could have a German tax system for the simplest filers. We got to the 1040EZ because we believe in theory governments should stay out of our business. Fundamentally this is a driver of the majority of the policies in America, and when viewed from the lens of other western countries it seems backwards because every country listed in comparison has a stronger, more involved, and (in my opinion) more dangerous government. Perhaps not dangerous now but given enough power and enough reason could become dangerous faster than America's current government system. In fact, the unparalleled level of power corporations in America have over things like tax law parallels the level of dangerous power governments have over their citizen's taxes elsewhere. It's an iteration on the same old process of control. Missing how they're the same it's simple to arrive at the conclusion America is the only "backwards" one. Usually this argument devolves into tax utility, which I won't get into here because that's a philosophical argument beyond the scope of the mocking of America that ALWAYS comes with this nonsense.
You're using 'how we arrived here' as a cheap excuse of an excuse to justify a status quo that is worse than it is in other places. That's the trope of American exceptionalism right there, to somehow find consolation in 3rd world conditions through repeated 'but we are god's own country, screw that even China has a higher life expectancy'. Really, the trope here is how the proud patriots of the richest, most powerful country in the world simultaneously feel superior to everyone, yet feel butthurt and threatened by essentially anything else on this planet that doesn't exactly act/think/look like they do.
> Sure we could have a German tax system for the simplest filers. We got to the 1040EZ because we believe in theory governments should stay out of our business.
That makes no sense whatsoever. As you write yourself, 1040EZ simply lets you confirm what the government already knows. The difference to the German system is that you still have to jump through hoops, roll over, and catch the ball when the government tells you to. I guess you also see the 'Obey the speed limit' signs in Texas as manifestation of supreme liberty, contrasted to the oppressive German 'no speed limit'.
Most other Western countries have much more powerful (= effective) checks and balances, as should have become exceedingly clear by the failure of the American system to keep an even openly criminal President and his attempted coup in check. That game is still not over. Most of Europe learnt that lesson by studying what went wrong in Germany 90 years ago.
But, I mean, you do you.
> As a software company, I can expense travel for some things, the most useful for me being conferences. But that means I need to buy the conference ticket even if I’m not interested and can only expense accommodations for the duration of the conference. That’s good but a bit restrictive.
> Now, as a artistic production company, I’m not limited by that. As long as I can prove I traveled in order to produce, well, just about anything that can be construed as being art, I can expense travel and I can expense accommodations for as long as I need. You can’t rush art, you know how it is.
Clever. But given the IRS just bought more guns and ammo than many small countries I'm sure this makes your dog nervous.
There seems to be a false idea that only the USA has tax loop holes. I assure you. It’s not the case. And tax heavens are also way easier to use as a EU citizen because there’s no global income declaration required.
I suppose it's also correspondingly easier to avoid tax Hell . . .
Your post was entertaining, but there might be a fly in the ointment! Are you actually carrying on a trade? Your consulting work is trade, but there are rules around businesses having multiple lines of business and what is and what is not considered a trade (see BIM20090 and possibly BIM85740). If your art is not commercially available and making more money than it "costs" in expenses, HMRC would probably determine that this line of business is a hobby with no expenses deductible.
If, however, you are selling this art, it's a fantastic wheeze, and also a quite legitimate one as you would, indeed, be a professional artist. You could, too, perhaps find ways to use said art in your trade such that it would commercially justify its creation, even at a high cost.
They’ve seen all these schemes before.
Why aren't there more employee-owned companies? And I don't mean solely shareholder programs, I mean actually owned by every employee and they are paid dividends of profit after all planned R&D costs. Similar to the Alaska oil pipeline bonus. Even interns.
All studies show that the companies structured this way that do exist (they seem few) have much higher output, quality, and happiness among staff.
After more research I found that this exists: https://esca.us/ - Wawa is a member, pretty cool.
Talking more to others about the idea I've heard interesting stories. Some machine tooling shops have an actual employee-owned setup. All employees are incentivized to make every product output great and keep profits high, because they all share the profits.
Anyway, I'm not actively working on this. But wanting to shape this idea more in the future. It feels like it could help the current state of America. But perhaps I'm too hopeful and naive :-)
---
Hobo Johnson - My Therapist:
"The idea's about equity, it's about wealth
Most think that it's dumb, you should think for yourself
If I buy a pizza place that makes a definite profit
Yeah, let's say yearly, the owners make $100,000 off it
And if I buy this pizza place for, let's say, $300k
And when the workers recoup in three years, I'll sign it over that day
And wouldn't everybody not see
They should buy their pies from me?
You'd rather have a boss
When you can work democratically?"
---
"Incentives are the strongest force in the world. They explain why good people do awful things, why smart people do stupid things, and why ordinary people do amazing things."
- Quote I have pinned from @morganhousel
How do you fundraise outside of the small pool of employee-owners?
There's actually a bill in congress right now that would, among other things, provide technical assistance and outreach for employees who might be able to use the program: https://www.congress.gov/bill/117th-congress/house-bill/4141...
My fear in that situation would be 'lazy investors' who donate some money but do no work to grow or manage the company.
so regular investors then xD
https://www.youtube.com/watch?v=15nberWl0EU
If the capital comes from the employees, this loss would be very hard for them and probably would not appropriate for more workers.
If the capital comes from a bank, is there collateral? If so, where did the capital for that collateral come from? If not, does the bank need to lend to you at an extremely high interest rate for it to be worth the investment?
If the capital comes from investors who have an equity stake in this, how is it different than any other VC? Who keeps the profits when the business is up and running (the employees or the investors)? If the employees, why would the investors invest?
Technically a co-op is usually "member-owned" whereas they were talking about employee owned. A substantial difference in some ways but not in others.
We are hiring software engineers, in case anyone is curious. https://www.teamshares.com/careers
https://en.m.wikipedia.org/wiki/Igalia
TL;DR There are many pieces to our culture and capitalist economy that make this difficult, but somewhat surprisingly to me, also many legal hurdles to formalized democratic corporate structures.
I've taken a few steps to talk to lawyers and contact advocacy/support groups for worker-owned co-ops and things, and there are a number of surprising hurdles for this type of formalized corporate structure.
In particular, whether a member is considered an "employee", an "investor", an "owner", or some combination of the three differs in jurisdictions, often at the _county_ level, not just the state and country level (experience primarily in the US).
US law gives a massive number of very specific rights to investors, making it difficult to pull someone's ownership shares if they leave the organization in some jurisdictions. Additionally, the tax implications for individuals can be extremely complicated.
Talking to a lawyer, they said one of their clients had been working for ~10 years to hire someone across state lines and hasn't managed to do it yet because of the legal complexity (some kind of employee-owned construction company in Northern California that wants workers from Oregon I think).
For the time being, my own structure is on hold, but I'm thinking of simply formalizing the structure in the equivalent of an employee handbook and putting legal ownership into some kind of blind trust then paying members at a rate tied to net income without formally specifying that it's a dividend, but I have no idea how many laws that might be inadvertently breaking.
In any case, Mondragon is an interesting case study, as they are (to my knowledge) the largest worker-owned cooperative in existence at the moment, but workers outside of their home country (France, I believe?) are not able to participate in ownership and profit sharing for legal reasons.
Another user posted about Teamshares (https://www.teamshares.com/). It appears to operate similarly: investors buy up companies and give them to the employees. I'm assuming some level of fee is given to Teamshares to continue operating.
Oddly enough.. one of the backers of Teamshares is Collab Fund, where my second quote is pulled from on one of their most recent posts. Small world.
As poster above mentioned fundraising is a major challenge. Seems like debt is your only option if you want to keep the thing 100% employee owned. Bootstrapped businesses are of course a thing and can scale to Unicorn level but at a much slower pace than a VC backed startup.
For about 8-9 years now my goal has been to build a cell from scratch for less than $1000 and enable anyone to be able to do so. While I’m handling the other bits elsewhere, DNA synthesis is a key technology primed for open source disruption.
“Your scientists were so preoccupied with whether or not they could, they didn't stop to think if they should.” —Dr. Ian Malcolm (Jurassic Park).
The logic doesn’t hold up.
You can make the same argument for anything. It makes sense to think about the dangers, not pretend that bad logic is an excuse to do anything you want.
The former tends to stop all progress, while the latter considers that negative outcomes are real, but that we can productively reduce risk.
Kind of like how OpenAI removed racist, sexist, and other troubling images from their AI training pool and then released it in pieces (to a small group of subjects) to remove as many issues as possible. With that said, they definitely still made mistakes.
It’s very easy to come up with answers to that question if you don’t know what is actually possible. Not very useful answers, though.
That is untrue. If you mean by "...it requires knowledge and expertise to answer" domain knowledge.
It is possible to understand the implications of technology without mastering it.
The experts in the field have huge incentives to say it is safe.
If we are to make decisions about releasing novel self replicating organisms into the environment, we should not task the people who will profit (scientists) to decide if it is a good idea or not.
In any case, blatant sarcasm is also out of the spirit of hackernew’s “ Be kind. Don't be snarky. Have curious conversation”
It is a bit dead when it comes to a git, since I am trying to get the first machine needed, which is a DNA synthesizer (asked about that here https://groups.google.com/g/diybio/c/V3OYVBxaH04 for example).
The idea is that with a traditional DNA synthesizer I can have positive controls of the chemistry, and develop a chip that can fit inside the flow cell of the existing synthesizer. In biotech, everything goes wrong a bit more often than in computer science, so the focus lately has been getting my hands on a working synthesizer. This is a tried and true method of getting chip synthesis working.
If that works, I'd like to provide the chip at cost for integrators, as well as develop a functioning full product for integration with some bots I'm building for my official work.
Personal website is here - http://keonigandall.com
Chisel for programming it (great little lang/project) https://www.amazon.com/Digital-Design-Chisel-Martin-Schoeber...
Efabless PDKs - https://efabless.com/
Other than that, lots of trial and error. Hardware people are brutal with their acronyms, so it really comes down to a lot of stumbling around and reading whatever I can.
Suppose nuclear weapons did not exist and someone was concerned about the Manhattan project — would you say that chairs enable people to build regular bombs, so we shouldn't worry about nuclear bombs?
One day, I will do it just because I can and because you argued I shouldn't. Then your actions will be a proximate cause of the existence of the thing.
Ideally, I would not be so controllable but a deep part of me loves the transgressive nature of fighting a puritan.
Upvotes for the bants
“Harmful” can also be seen the other way. From my view within the industry, it is harmful that the tech does not exist, and the idea of harmfulness is, largely, propaganda perpetuated by elites to maintain the status quo.
I am pretty new to chip design but know the DNA space quite well - I would love to do a call, if possible, because I have lots of noob questions about chip design and chip design tools. Happy to share anything about the biological side in exchange!
Is it something like a 3D printer where you can get a rough but usable finished product, or is it more like a "if you want to make bacterium X produce Y product, you need to do X, Y, Z, W, etc." and the chip is used to do one of the steps.
Any resources and books you'd recommend?
https://www.zerotoasiccourse.com
And also:
https://www.udemy.com/user/anagha/
Efabless's chip ignite program is also great - check it out! https://efabless.com
_____
How to make DNA synthesis affordable
https://keonigandall.com/posts/affordable_dna_2.html
I encourage him to continue explore new ways of making DNA, but understanding the market is very different from building off an expired patent.
I would like to know what specifically you think I am wrong about, though. Always happy to improve my thinking.
If so, why would one want to do that versus taking an existing cell and injecting custom DNA?
Will the output of your product add a suicide timer to a cell?
Will the output of your product prevent the cells from procreating/multiplying?
Will the output of your product prevent pathogen creation?
Will the output of your product require a specific, unnatural energy source that can only be man made?
Professionals take great care in thinking about those problems, and sometimes still fail. (IIRC, a synthesized breed of mosquitos that were released in Brazil failed to die off and are now a part of the biosystem).
[0] https://www.dw.com/en/genetically-modified-mosquitoes-breed-...
How would unicellular organisms procreate exactly?!
I am not being pedantic here but procreation usually entails sexual reproduction and I don't see how this is possible for these organisms.
1) Multiplying: Asexual reproduction only.
2) Reproduction: Sexual and asexual reproduction.
3) Procreation: Sexual reproduction only.
BTW: I don't see how 'multiplying' could refer to just asexual reproduction. People often use it in sentences like 'the deer population near here multiplied the last couple years'.
As you can see, getting terminology right is challenging and not always straightforward.
Perhaps part of the disconnect here is not realising how vast the applications are for synthesizing DNA oligos are. It's a very basic thing, and anybody can already basically order them online for a very affordable amount. It's like being worried about someone open sourcing a way to make printer ink.
It’s interesting how people revise their arguments, or omit words from it, to make their interlocutor’s response appear more absurd.
Err, wait, ummm, it looks like we first need a R.E.C.U.R.S.I.O.N. to establish the R.E.C.U.R.S.I.O.N.
It's super hard, but until novel bioweapons are discovered, it is at least a tractable problem.
Absolutely not.
> Will the output of your product prevent the cells from procreating/multiplying?
Absolutely not.
> Will the output of your product prevent pathogen creation?
Strictly defined as the output being a chip, absolutely does not.
> Will the output of your product require a specific, unnatural energy source that can only be man made?
Absolutely not.
> Professionals take great care in thinking about those problems
I am a professional in this field, and have been thinking about these problems quite deeply (if you check on my website, my first time writing about my concern for these problems was back in 2014). I have developed opinions on this over the years, but roughly they come down to the fact that many folks have a gross misunderstanding of the field in general, but quite like to think that they understand what is going on.
For example, I mentioned I wanted to do oligo pool synthesis - how the hell would the output of an oligo pool synth run add a suicide timer? Or prevent replication? Or require a certain kind of energy source? In the context of the stated goal, these objections really don't make any sense. It is roughly equivalent to someone wanting to run a mining company and getting countered by "will the output of your product stop school shootings?". Perhaps better questions are along the lines of - how are oligos matching biohazard sequences prevented from being synthesized? Well, this is a question of both governmental policy (what IS a biohazardous sequence?) and of the integrated device (does it phone home for each synthesized sequence? What about hardware hacking?).
> Will the output of your product prevent the cells from procreating/multiplying?
Why would this matter? We have immortal cell lines like hela cells that have been alive for decades.
> Will the output of your product prevent pathogen creation?
No, but no one is going to "accidentally" create a new pathogen (you'd need this as well as some expensive labware and a lot of expertise), and the people with the incentive can already do so in labs.
> Will the output of your product require a specific, unnatural energy source that can only be man made?
What lol
It’s incredibly interesting and very niche. There are a handful of academic papers out there. I think it will be a big deal in the next 10 years.
“Expert” is that you’re asking people who should know what they’re talking about, not random laypeople.
“Elicitation” is that there’s a whole process to it. You don’t just ask “What are the odds this movie will do $100 million in domestic box office in its first month?”. You ask lots of smaller questions and use visuals to build up an answer.
I also like how “expert elicitation” sounds like we should be doing a good job eliciting!
We already have so many places internally to put in forecasts. Cash flow projections for example. But, all you get is a text box. I’m working on making that easier and more representative before anything else. Making a web app (when all you have is a hammer…) to start. From there, there will be lots more to do but beyond what I’d like to share publicly for now.
[1] https://shelf.sites.sheffield.ac.uk/
For years I've had this idea rattling around in my head to build a web UI that shows a graph of some historical quantity, e.g., company revenue, disease cases, global mean temperature anomaly, stock prices, etc. The historical data is shown as a curve which terminates at the present, and there's some empty space for the future. The user then uses a freehand drawing tool to extend the historical curve into the future, based on what they feel will happen. Save this data, and let users view some average(s) of all crowdsourced curves, grouped by submission date. Stretch goal: let users track their own forecast accuracy, and let users view forecasts weighted or filtered by the historical forecast accuracy of contributing users.
Is this close to what you're working on? I have no idea how useful or interesting this would be, but I feel like I'd learn a ton just trying to build it. Alas, I just haven't made the time, and lack the skills to hack something together in a reasonable time.
ex. https://goodjudgment.com/
Ps : I knew today's Standard technology was 360°photos. But I was convinced this was not an immersive enough technology.
Immersion, for me, should feel more like a video game FPS.
So I gave it a try: - A No-Coding Editor, just photos as input, plus 20 minutes of clicks and you have your immersion.
Solo project
The structure it currently takes - academic journals - made perfect sense in the 1600s when that structure was developed, and it continued to work reasonably well for distributing academic results up into the early 1900s. But then it got privatized. Now there are tens of thousands of academic journals, 80% of them charge a fee for access, and most of the remainder charge a fee to publish. Often thousands of dollars.
Given that science works in the aggregate and you can't know if you have the real answer to your question until you've accessed all of the literature on the topic, this structure is now making it impossible for people not in the institution to even figure out what we know on a topic. And hard even for people in the institution.
The ultimate decider of policy in a democracy is the average citizen. The people who decide our government policy don't have access to the primary source material of science. In the US, a lot of really important policy is set at the municipal level (city, county), and in that case the people actually writing and implementing the policy also don't have access.
If we can develop a web platform that does all the journals do (match papers to qualified reviewers, maintain literature integrity by filtering out bad work, and dole out reputation), then I think it's possible we could draw publishing out of the journals and into the open where it belongs.
I've got an idea for a platform that does all that by crowdsourcing it (the journals are already crowdsourcing it - just manually using an editor). It's basically Github+StackExchange for academic publishing. It works by tying the reputation system to academic fields, so papers are tagged with fields and then reputation is gained and lost in those fields.
I'm building it now, I'm month or two out from beginning a closed beta. (Aiming for end of October, beginning of November.)
I wrote up a detailed description here: https://blog.peer-review.io/we-might-have-a-way-to-fix-scien...
They operate mostly in the computer science academic community, last time I checked.
For example, there are several attempts to overlay review on pre-prints and open repositories (like Zenodo), but they aren't identifying qualified reviewers and matching them to papers in any way. There are the repositories and pre-prints themselves, but they aren't providing review. There are attempts to build open journals, but they're still taking the journal form, which means they still have lots of manual overhead.
Most of them have varying amounts of traction, but few seem to actually be on a course to replace the whole system.
...it remains to be seen whether peer-review.io will fare any better. I think it has a chance because it does provide an alternative system for each of the services the journals provide. Well -- except one, which is the moderator role the editor sometimes plays. That may prove fatal, but only time will tell.
Since we're on HN, we get to take off our technical hat and put on our product hat -- not the fit-to-customer product, but disrupt-the-industry product.
What are the incentive structures that would destroy this or make it work?
Consider the scenarios today...
- What's wrong with the thousands of similar-sounding journals popping up on China publishing research that is either unvetted or copied from other journals, so researchers can satisfy their publish-or-perish needs?
- How do you deal with reviewer networks?
- How do you get the best of the best, people with no time and plenty of opportunities and money, to provide effective feedback on an avalanche of articles?
- How do you get the support of, without being dominated by, the big research companies and universities?
- Look at the history of reputation systems, from quora to stackexchange to games, on a matrix with the difficulty of determining correctness of questions:
-- Who won the game?
-- Does this code work?
-- Which library should I use?
-- Is this paper correct and valuable?
- What about prediction markets, which try to use money/investment as a measure of seriousness: how would that help or hurt? (and isn't that what we're doing by insisting academics publish for tenure?)
In the spirit of MVP, you might consider a pivot to a much smaller problem: how to professional groups establish and document their standard of care? A team wiki? (gets stale, disconnected from operations) Run books? (too stepwise to convey meaning needed to transform the system) Slack-capture? email? documentation?
What about capturing the benefit in a change of format, from opaque text/pdf to something like a per-domain semantic web, with connections to empirical methods and findings? That could provide incentives for everyone. Like the professional-group scenario, the killer feature would be something that grows incrementally with multiple authors (as code does for developers).
(Although, the first contribution would probably need to be getting the local working again in a new context... I've been going fast and taking on some techdebt that will need to be paid down soon.)
Automated reviewer matching won’t be good enough to find decent reviewers.
I would suggest partnering with people in very specific disciplines who want to break away and establish open access journals, but want to focus on the editor and review process rather than the logistics (which you provide).
I suck at naming - Peer Review was a working title intended to be somewhat tongue in cheek (similar to StackOverflow). But... because I suck at naming, it is, of course, a little on the nose. So far I haven't come up with anything better.
To toss a few names in the hat, off the top of my head: - openresearch - colab - openjournal - stud.io
Probably collect a few ideas and run a small survey for the initial audience
If the authors disagree with the journals team, or just get tired of waiting, they can still publish at any time just like with community review. For authors who choose both, the journal's team and community reviewers can interact with each other just as in normal community review.
I think it's a way to provide a stepping stone for people from the existing system towards full community review and would provide logistics to those organizing open journals with teams of high quality reviewers.
The plan is to implement it during the closed beta period. I would welcome feedback on this concept as well :)
May I also suggest you consider the network layer centralization. When you mentioned Github+Stackoverflow, I got the point even before I visited your site.
However, even as you think about an alternative on how we publish, consider that technical questions can have significant political consequences. I am of the view that centralized networks are a major contribution to the situation we find ourselves in today. Distributed/Decentralized/Federated options like ActivityPub may help in your journey in what surely is a great idea. Check Lemmy, for example, on a real stackoverflow option - https://join-lemmy.org/ and Gitea already working on a federated "Github".
Like if a major paper is withdrawn, or a theory disproven, could you get a list of significantly impacted papers?
Often they do - quite a lot of stuff is open access. The biggest issue with getting people to read scientific papers is the demonization of those who actually do it. A lot of papers are terribly misleading so the moment outsiders start engaging with the literature they conclude a lot of science is junk, and when they act on that they're immediately attacked as "science deniers" ... by the sort of people who don't read papers, because they are convinced they don't need to.
Getting more stuff as open access will only accelerate the decline in trust in science, by allowing more people to see what's going on behind the curtain. That is not necessarily an issue, but just so you're aware of that - it isn't going to lead to lots of people suddenly basing policy on scientific papers. It's going to lead to a lot of scientists getting defunded.
The natural-language plaintext is fed into a language model that annotates it. That output is converted into an intermediate representation based on relational algebra, which we manipulate and finally compile to SQL.
It's probably not scientifically groundbreaking, but there are a lot of interesting technical bits along the way, at least for me.
On the other hand we are aiming for reasonable response times on large scale datasets.
(One thing I want to emphasize here is how abysmally, unbelievably, depressingly god-awful the vast majority of database schemas are.)
You wouldn't expect any user-friendly feature to be automatically stamped on top of a database without the involvement of a programmer, and this is no different. You need some sort of descriptive layer between the database and the query engine (or beside it, helping out) that programmers are involved in building. AI is not going to magically solve this problem for you. Figure out what you need described to you and turn that into a meta language that application developers can write to help your engine out.
Frankly I'm wary of any direct link between the user and the database. How do you deal with constraints like "users should only be able to see data associated with their organization in tables X, Y, and Z" if the programmer doesn't tell you? How could you possibly prevent exploits and security problems? Another reason to focus on querying an intermediate layer or relying on a programmer-provided description of what's available.
(Or am I assuming way too much, and "hypotheses on the schema" is already this meta-language?)
By "hypotheses on the schema" I mean we are assuming the schema is already in a very normalized form. I fully agree most database schemas are awful, this is a deliberate choice to avoid getting us into that kind of trouble.
The "intermediate layer" role is played by the IR I was talking about in my previous comment. The user would never be able to see anything not provided by the IR because it's not even representable.
Our scope is much narrower, really.
Without consideration for mobile where IPs rapidly change currently it’s based upon IP swapping and I recently added a convention to auto-update addresses when a node makes a connection to other trusted nodes.
Later I envision an optional opt-in service that resolves node identity to IP address. Something like DNS but for hashes instead of domains.
Currently I am focused on updating a bunch of test automation and then I want to turn my attention to adding a command shell to the application GUI, which would solve for SSH into remote personal devices in your node list.
https://deliberati.io/give-truth-the-advantage/
I was pretty late to GME, despite having been on Reddit for a good few years. I started researching what had happened, and where the conversations were taking place just trying to understand where to place myself to be a part of the next one. I ended up checking out the typical r/WallStreeBets and the other big trading subreddits but I always felt that I'd need to devote 100% of my time to Reddit in order to capitalise on any of the information. So I built a scraper to do the heavy lifting for me. That worked well. Then I started trying to find ways to identify the good stocks from the bad. That worked even better. So I turned it loose onto many different social media platforms where people discuss trading.
It worked well enough that I ended up quitting my job to develop it and trade full time. There are still improvements that I'm working on but I think that'll always be the case.
A great proof of concept of this is BBBY last month. Out of nowhere it went from ~$5 to $30. There was no real business reason for this, just rumour and speculation. It won't be the last.
I don't know if links are allowed but if you want to follow along, it's https://feetr.io (unfortunately not accepting any more beta users currently, but all data is posted to Twitter until we launch). There's also a leaderboard at https://feetr.io/leaderboard if you're only interested in the big numbers.
The tech stack is 100% lisp. I use CCL locally, SBCL on AWS. Also, hunchentoot will power the backend when it launches. I mention that because a lot of people have this fear that hunchentoot is slow or something but in my case it's pretty rapid. Of course, make sure you're writing fast code, but it's not the bottleneck that I think people think it is.
In the meantime, someone will be making money from the stock market. Why shouldn't it be you? Or me? Or anyone reading this?
Ethics is hard because there's no enforcement like there is in the legal system. You have to think through the consequences of your actions yourself to understand the harms you engender throughout the system you inhabit, and then choose for yourself what level of harm you feel is acceptable to bring onto others for your own benefit.
It would be amazing if we lived in a world where morality would win out but we don't. My concern is "how do I help the most amount of people?", and that's why Feetr is public and cheap.
You can live the life that makes you happiest but how are you helping people deal with the current economic downturn? There's a food bank near me and the line is much longer than I'd care to describe. It's not even cold yet. And it's going to get much worse.
You can either pretend to help by avoiding the big bad, or you can work within the current framework to try to do as much good as you can.
Further, why do you imply that participating in the market contributes to "good"?
Your reliance on a false dichotomy to make your point shows everything about how little you care about the actual problems we face as a society, if you can only come up with such unimaginative and ultimately selfish approaches.
I don't understand why people snap back like you have, when you have literally nothing to stand for.
I had a friend that did something for crypto. He would try to map pump and dump efforts to crypto projects. Gave them codenames and followed which ones had the most successful pump and dumps and then get in on them and sell early. It was a lot of work getting the spam early and being involved in all the places. I don't think he ever managed to fully automate it.
It's a manual process of discovery. As in, I actually go out and hunt down groups talking about stocks and if it looks interesting, I'll add it into the algorithm. I'll then analyse the data over a period of a week or two and determine whether it's worth continuing. If it is, I move it to production and it'll perform as the rest do.
If a community is not publicly visible (discord, telegram, etc) I'll ask an admin if they're okay with what I'm trying to do. If it's a no, it's a no.
There are those groups that do try and force a pump and dump but I'm against using them. That's ultimately not what I'm after as I prefer organic conversation with genuine reasons to buy. Maybe the reason doesn't turn out to be true (BBBY) but that's the case with a lot of stock analysis anyway. Sometimes the reason a stock makes it to me is due to a pump and dump but there are measures taken to make sure that we're not acting on artificial metrics.
There is also a period of premarket validation which tries to measure the stocks that are deemed interesting each day, so people are (hopefully) not getting sent duds. It can happen, yesterday ADTX only managed 0.58% but moments like that help us fine tune the algo and those happen fewer and fewer.
I'm not sure what you're trying to do now. The most interesting (for me anyway) would be to figure out what is going to be on Reddit before it appears by analyzing spam/pump and dump groups etc. If you're analyzing Reddit after the fact, isn't that kinda late?
It might be late in that you might make less money than you would if you were in those groups but I can't say that I'm unhappy with what the algorithm is currently finding. It's primarily built for day trading, so you're in and out of trades that same day. There are times you're out within 30 minutes. The goal is near daily compounding, which leads to higher gains than holding for X number of days.
It has been recording data since 9th of August 2021 and is averaging 4.5648% across 340 stocks. Since Jan 1st this year, it's averaging 5.1779%, and I would like to think that I can get that number higher.
Note that these are perfect values, using the open price and the highest price of the day, they're unachievable consistently, I use them to understand how much potential for profit we could've had.
I guess what you call "organic", I call the very end of the tail of a "pump". I believe that Reddit sentiment is largely manipulated. When GME was going crazy, there was definitely a coordinated effort to pump AMC, BB and some other garbage that I forget.
I don't believe that suddenly a group of people will all randomly pick the same tickers to start hyping up. I'm not saying everyone involved is trying to pump/dump those stocks, I think it's a small group of people posting bullshit analysis in random places trying to get other people to buy + hype up the stock to create that "organic" interest that you are capturing.
I reckon BBBY was definitely an organized pump and dump for example, but only a small amount of threads/comments were organized. A lot of people buy into the hype and continue to hype it up themselves.
The most interesting part is identifying which stocks will end up "going" viral, by analyzing who/when/how the first mention of these tickers starts up. Maybe a certain writing style works really well, maybe if some ticker gets mentioned at the same time in 3 specific discord channels then that stock goes on to do well. Maybe if a certain reddit account posts it first it goes well, etc.
Using PRAW to get counts of ticker mentions in some subreddits isn't super interesting, at least in my eyes.
Just to clarify, this isn't what is happening. And I agree that's not interesting. It's measuring the response to mentions and that's the score we save (we call them impressions). It's like that saying "if a tree falls in the forest, does it make a noise?", if a post goes unseen, will people still buy the stock?
We also don't just release the top X number of stocks that we recorded per day, we're looking for patterns in the data and this becomes the interesting stocks of each day, which then go on to be validated.
> that suddenly a group of people will all randomly pick the same tickers to start hyping up
Of course not, like you said: it's memetic. Sure it can start off coordinated but when it goes viral its self-perpetuating. Those are the kinds of patterns I mention in the last paragraph. You can see these upticks, the strength of them, and try to gauge how sustainable they are.
Here's me semi freaking out about BBBY just before it really ran:
https://twitter.com/0xsmcn/status/1558770400095502336
Is it weekly/monthly/yearly? Are you tracking buy-sell for each stock daily? Are you taking into consideration loss potential? One loss could wipe multiple days gain.
The stocks increase by that daily. You will not reach that value consistently, I urge everyone to take a conservative approach when investing, and to take profit whenever they can.
We don't (currently) invest peoples money for them, so we need a benchmark to understand how well the algorithm performed, and that was deemed the fairest.
- Performance: How to scan files as fast as possible?
- Tag-Storage: How to store audio tags as generic as possible?
- File-Watchers: How to prevent fully indexing the filesystem over and over again and only react to changes?
- File-Sources: How to manage multiple file scanners as efficient as possible?
- Cancellation: How to cancel running tasks, when File-Sources are removed?
- Moved files: How to not lose all customized information (rating, playlists, playcount, etc.) when a file gets moved to another place?
- and many more :-)
[1] https://github.com/sandreas/tonehub
If you have a worker thread that has a hot loop that checks a pthread structure as its limit. You can then send a virtual interrupt to it, or preempt it from running further by setting the loop to its limit.
You can see my C, Java and Rust code of this in this repository: https://github.com/samsquire/preemptible-thread
It can be used to create extremely responsive software without slowing down the system with cancellation checks.
https://github.com/samsquire/ideas4#120-cancellation-trees https://github.com/samsquire/ideas4#99-register-loop
One problem I have with computer systems is data liveness and synchronization. You want to react to change in many situations but don't want to do things inefficiently such as polling regularly.
You kind of want to react to change given an event, when that event happens. So you don't need to poll and compare.
You also have the problem of identity, how you map data to other data and keep it in synchronization.
If you can capture events at source, then you could do the right behaviour. But it's very hard to capture events at source in modern computing systems as not every API has a callback or event log mechanism.
The first milestone will be providing a basic API for my files - the main components of this will be the database (postgres), the API (C# + swashbuckle + JsonApiDotNet + Websockets) and the file indexer (C# HostedService). All parts except the file indexer are pretty much done, but it is a critical component, because it has to be as fast and correct as possible.
There are multiple approaches to index files... A best case scenario would be an "import" / "move" of files into a library or repository. That way you would be always up to date and always perfectly sorted. Unfortunately, an import would also be a big amount of work, because analysing the files and getting metadata from online sources is... lets say a huge project. And NOT getting metadata would mean, that I cannot move the files while another app manages the metadata. So I took another path - scanning an existing and well tagged library (that I manage with beets[3] for music and m4b-tool[4] / tone[5] for audio books).
My current Idea is to have a file indexer that:
- can run on multiple sources
- runs one full index scan after starting the app
- registers a filesystem watcher for every file source and reacts to events
- To ensure, no filesource is blocking others, each source is processed by a fixed batch size and then move on the the next file source
- If sources are modified (added, changed, deleted), there is a decision, what to do with already running indexers and registered file watchers (added just go to the queue, changed and deleted cancel already running tasks only for this source)
- All files are hashed (content only) to ensure, a change of metadata or tags will not change the hash, and if a file is moved, it will recognize this and update instead of delete and insert
The database will contain Tag-Values for every possible value. E.g.
That way I can add a fulltext index on the Tags.Value field containing a searchable value while maintaining the FileTag.Type for recommendations.Let's say I search for `AC/DC`, it will provide an auto-complete for all FileTags.Type values, that show a match + a generic one for searching ALL values:
Searching for 2010 will show: because it contains matches in Releasedate and title.There may be a lot to optimize, but I think my current plan goes pretty well. Let me know what you think about this approach :-)
[1]: https://www.navidrome.org/ [2]: https://www.audiobookshelf.org/ [3]: https://beets.io/ [4]: https://github.com/sandreas/m4b-tool/ [5]: https://github.com/sandreas/tone/
I am guessing you want to keep the code that handles file events on the watcher and on startup the same code used in two places.
Guessing you scan multiple source directories of files recursively.
Does C# have a thread safe queue object? You could create a pool of worker threads and the file watcher can enqueue events
You could have threads that scan file sources (one per source) which enqueue file names to worker threads which do the work. You could have a queue per source thread and worker thread.
The problem with the file watcher code is that I don't know what context that event runs in, so you would either have to enqueue events from the main thread context to one of the worker thread queues.
Yes, Producer Consumer pattern. Currently a single thread each, but that would be scalable later. For now I try to keep things simple.
> Guessing you scan multiple source directories of files recursively. Does C# have a thread safe queue object? You could create a pool of worker threads and the file watcher can enqueue events
Yes. There are a few. I use BufferBlock<T> [1], which is pretty flexible.
> The problem with the file watcher code is that I don't know what context that event runs in, so you would either have to enqueue events from the main thread context to one of the worker thread queues.
This is the long term plan. Using events is much more flexible than "polling" the next batch of file items (even if it is in realtime). The architechture seems to work out for this but I think for now I'm pretty close to a working solution. Maybe I start going for it, develop a small UI in flutter and see, where there might be problems :-) Currently there is too much "theory" - I would like to see this in practise.
What do you think?
[1]: https://learn.microsoft.com/en-us/dotnet/standard/parallel-p...
You're working on something interesting.
Thank you for sharing!
Many filesystems try to solve the same problem (eg: customize the appearance of files in a particular folder [1]). One solution is adding extended file attributes [2], however this might not be supported on all operating systems.
[1] https://en.wikipedia.org/wiki/.DS_Store
[2] https://en.wikipedia.org/wiki/Extended_file_attributes
A slower but more portable solution might be content-addressable storage. Basically, create a directory containing just metadata files for each song. Name each file as the SHA256 sum of the associated music file, and put metadata into it in a binary format like flatbuffers [3] or Cap'n Proto [4] or a plaintext format like TOML [5] if you prefer to make the system human-editable at the cost of lower performance. Even after moving a file to another location, the SHA256 sum of the file should not change.
Note that if you have duplicated files, then there might be hash collisions where you'll have to reconcile metadata differences (or you can just merge the metadata together, keeping attributes with the later timestamp). There are various solutions to this as well like building a parallel directory structure which mirrors your music filesystem, but that can get complicated.
[3] https://google.github.io/flatbuffers/
[4] https://capnproto.org/
[5] https://toml.io/en/
- File-Watchers: How to prevent fully indexing the filesystem over and over again and only react to changes?
When first loading a directory of music into the program, build a merkle tree [6] of the files' hashes and save them to the content-addressable storage directory described above if they do not already exist. Once indexing is complete, serialize the merkle trees for each directory as well, this way the next time the program starts, you can just load these up and check for consistency of the files in the background. Then set up FileSystemWatcher [7] to notify you when the contents of a directory changes, and update the metadata files and merkle trees accordingly.
[6] https://en.wikipedia.org/wiki/Merkle_tree
[7] https://stackoverflow.com/questions/721714/notification-when...
My current solution / plan is:
- Get byte offsets of the audio stream only (e.g. 384, 834882948) - Ignore metadata and build xxhash only over this part - To be faster, build the hash over 5MB in the center - If a new file is indexed, build hash and look it up in the database - If one match is found, assume this is the file - If 0 or more than one, assume new file
The edge case of 2 or more with the same hash has never happened. Worked out pretty well so far, although its not perfect.
And C# File API provides FileWatcher possibility, so I get notified whenever there is a change. The first run has to be done as full scan, after starting the full scan, a FileWatcher is registered and pushes to the Queue whenever a file has changed.
It's far from perfect, but it works.
There are various other big dictionaries and websites specific to different dialects, but I don’t want to have to search on each of them by hand.
I’m currently working on a meta-dictionary, which is a fancy name for unique search engine for all the dictionaries I can find out there. It’s not finished but so far it works great; I use it almost daily.
I started with Phoenix (Elixir framework) and Svelte to learn them but I ditched Phoenix because it was a hell to work with how implicit things are there (no import; just assume that variable is defined "somewhere") and went back to a very basic Python Flask app that serves an API for Svelte.
Nothing very fancy on the tech side but it’s interesting to deep dive in the languages of Italy and learn a lot of things about both Italy and linguistics at the same time.
https://github.com/Dushistov/sdcv https://github.com/BoboTiG/ebook-reader-dict
Do you find this issue persists even if you attempt to search the Italian internet for definitions/usage examples in Italian?
Or is this problem specific to finding information in English about Italian terms?
The difference here is between an n problem and an n^2 problem (if the site expands and wants to accumulate definitions from each language/dialect to ~several supported major languages, for example).
Anecdotally, I've found that once I got to the point where I could read Italian well enough to use Italian dictionaries when I don't understand the word, finding the information on the Italian internet has been much easier. Of course, certainly there have been dialect words and slang I haven't found (which I attribute to some of the words being in a dialect mostly spoken by older Italians who don't use the internet).
> The difference here is between an n problem and an n^2 problem (if the site expands and wants to accumulate definitions from each language/dialect to ~several supported major languages, for example).
Yeah; here it’s not an issue because I only search for Italian dictionaries, I voluntarily exclude any sort of Italian<->French/English/etc dictionary because I learn a lot more when I stay in one language instead of translating.
> (which I attribute to some of the words being in a dialect mostly spoken by older Italians who don't use the internet)
Yes. Even in the case of Neapolitan, which is still very active, it seems that all the studies of the language were done before the Web, and so you find a lot of good paper dictionaries in Naples, but pretty much nothing online. Most of the content is found on some random blog where someone listed 300 verbs in Neapolitan or an AltaVista page where someone wrote the meaning of a couple hundred words.
I started to teach myself unity in December 2021. I've personally experienced benefits to writing automated tests and using CICD; therefore, I thought it would be fun to learn about writing tests for a 3D based software. It will be different from the web/CLI based stuff I usually write.
So, it turns out that the video game community (or at least the online circles I frequent) are extremely against the idea of writing automated testing for various reasons. This translates into there effectively being a non-existent pedagogy around teaching how to write tests for 3D based software. Content is scarce, and the content you do find is produced by people who obviously don't write automated tests. So, for me I've hit the books to arbitrage & translate techniques and philosophies into this "untapped" domain.
It's interesting because I've figured out tips & tricks that I'd consider low hanging fruit... For example, if you place a "test" camera in the test case, you can actually see what's going on in the test when it's executed. Or how important it is to clean up every created game object in your test after each test; if done right, you can keep your SUT at origin (0,0,0). Or the importance of "test prefabs" who are effectively mocks of other "real" prefabs...
One innovation that I would like to use/build that I truly consider (((revolutionary))) is this: I want the test cases I write to also automatically (or when tagged with a certain C# attribute) generate the same game objects in a "exploratory(manual) test scene." I think this innovation is the "killer app" that will completely and totally sell the value proposition of automated tests to those opposed. If you decide to build this, please also publish it on openupm and reach out to me.
I have plenty of other thoughts and ideas on this space. I love talking about automated testing; it feels like a very futuristic programmer practice. I hope this post demonstrates why this work and space is so interesting.
The client had a secret level with all game objects for testing and debugging. But reading your post makes me see that we didn’t think ahead far enough: the secret level could have been the foundation for automated unit testing game objects and battling characters.
I haven’t watched them yet, but I added these relevant videos to my YouTube backlog just this week:
* Automated Testing of Gameplay Features in 'Sea of Thieves' (GDC 2019): https://youtu.be/X673tOi8pU8
* Taking CI and automated testing seriously (2018): https://youtu.be/YGIvWT-NBHk
But I'm not dismissing testing in unity. I think there are some simple approaches to testing automation that could be done. It would be nice if unit testing made more sense but in reflection i've often found it's the integration of components that breaks or changes, less than the behavior itself.
It's a lot more technically challenging than we thought. For example, what's the best way to statistically guage a student's knowledge level? How do we recognize when an ngram counts as a new concept?
We haven't launched publically yet, but we have a LinkedIn page of you'd like to see when we do. https://www.linkedin.com/company/conceptionary/
>> trying to automate tutoring.
>> what's the best way to statistically guage a student's knowledge level?
As you have probably already discovered, there is quite a bit of research in this area :). If anyone else is interested, you can search Google Scholar for "deep knowledge tracing", "bayesian knowledge tracing", "performance factor analysis", "knowledge space theory", "intelligent tutoring systems", and "item response theory". Some intelligent tutoring systems include ALEKS, Squirrel Ai, and Riiid.
I guess it's pretty common practice for corporations or private equity to snatch up smaller independent practices and then run them like a sweat shop. Her current CEO's resume is a list of Starbucks, Walmart, and a few other corporate roles that are totally unrelated to the medical industry.
We're tinkering with a platform that can somehow disrupt this cycle and add some transparency to the practice structure for the would-be employee.
The current form is kind of a niche Glassdoor, or we've been thinking of it as the "online dating between veterinarians & practices." Still playing with the concept.
Anyone have any thoughts on how to take out these massive PE firms that are plaguing the Vet Med industry?
Here's some more info on the problem: https://www.ftc.gov/news-events/news/press-releases/2022/06/...
I'm convinced Vetmed will follow human medicine's history. Currently, the future of human medicine is small clinic subscription medicine without insurance. Apps like Roo could help with temporary staffing issues.
Let's talk: https://www.linkedin.com/in/nicholascgilpin
Investors can only invest in scalable, replicable businesses. Veterinary practices are intensely personal, often specialize in different species and breeds, and can develop unique experimental treatments. The cost of labor is ridiculously cheap (people become Vet Tech's for love, not money), and buildings need not be in commercial centers, so capital costs are minimal.
Honestly, veterinarians are really in the best possible profession for maintaining independent practices. If they can't resist investor takeover, who can?
This is absolutely not true and it's a big issue in the field currently (my wife is a Veterinarian). This kind of thinking is also what is causing a lot of mental health issues in the field.
That is what burns the labour out
I guess the question is then whether labour is abundant/available. Even if there are people willing to work for little, are there anywhere near enough?
Perhaps they should start a union, like the rest of us?
Also curious on what your source is for their high margins since you mentioned you're an outsider?
Have you considered forming / joining a union?
Over the past couple of years, the popular search engines haven't given me great results, and I've had to use a series of different engines, search operators, and dorks to prod out the search results I'm looking for, with mixed success. The front ends have also become a bit too feature rich for my use cases; I prefer a minimal front end with just search results, but most front ends have links to news, videos, drop downs, cards, cookie banners, SEO spam, and have too much javascript. Too much time spent loading and rendering the page, and too many requests, in my opinion. Instead, my search engine only take 1 HTTP request per search result page and is only a few KB in size.
Right now this project is still in the early stages, and mostly proxies bing's api with some minor adjustments, but if anyone is interested in testing it out here is the link to the search engine [0] and a link to the blog with an RSS feed to follow future updates [1].
[0] https://simplesearch.org
[1] https://simplesearch.org/blog.html
(And yes, realize yours is currently basically just custom version of Bing.)
The architecture allows any value to be mapped to one or more keys and any key to be mapped to one or more values (unless an attribute on the data object prevents it). These KV stores can be used to attach tags to other objects. They can be used to form columns in relational tables (i.e. a columnar store). It can be used to create indexes into file contents.
Each object is designed for parallel access by multiple threads. Only a single block needs to be locked to update any value or key so multiple writes can occur at the same time. The data in each object is organized to find values or keys without inspecting every block. It is very fast and allows both OLTP and OLAP operations on the same data set.
So far, I have used them to attach tags to millions of files and do searches. I have created relational tables that I can query much faster than the same table in Postgres (https://www.youtube.com/watch?v=OVICKCkWMZE). I have created 3D relational tables that can be queried the same way 2D tables have traditionally been queried (https://www.youtube.com/watch?v=AXqvWMmoL1M).
The software is currently in an open beta and anyone can download and try it out on their own data set. https://www.Didgets.com
If the KV store is a 'state' column in a DB table of US customer addresses for example, the number of unique values is limited to 50 (if you ignore D.C., Puerto Rico, etc.). The table might have 100M customers, each mapped to at least one state. Each state (except less populated ones like Wyoming) might have millions of rows mapped to it. Many customers might have addresses in more than one state.
The system has to do things like: "Find every key mapped to a state that starts with the letter 'M'" or "Find every value that is mapped to this set of keys". These are the kinds of things a normal SQL query has to do.
"pet":"cat" and "program":"cat" have different semantics.
So some typical queries might be: Find all the photos where "Event" = "Wedding" Find all rows where "State" = "California"
The keys in this case would be the IDs of the photos or the row keys in the relational table.
Generate a planar graph, preferably without a Delaunay triangulation.
One was an indoor web map built on an entirely open-source stack (QGIS and OpenLayers): https://map.fieldmuseum.org/ It was an interesting challenge because the open-source web mapping libraries (and many/most of the commercial ones too) were designed for outdoor use and didn't really account for indoor idiosyncrasies. Take rooms and doors, for example... while you can import vector SVGs geoJSONs to these mapping libraries, you can't easily indicate "this shape is solid except for these two doors" -- whether for visuals or for an eventual turn-by-turn routing/navigation layer. Another challenge was the concept of "floors", overlapped groups of geometry (walls, balconies, whatever) and points of interest that are mutually exclusive. We ended up having to trace floors against each other in QGIS (map editing software) and then adding state at the Javascript level to store each floor as a separate object to render or hide. Then there were a bunch of UX/UI tweaks we had to do, like how to draw arrows for Covid one-way flows, how zooming should work (we categorize POIs according to level of importance, then show/hide them at different zooms, and also dynamically scale fonts to ensure readability), which geometries to make clickable (a sidebar opens with pics and more details), how to hide easter eggs.
The whole thing was written as a vanilla JS single-page app tied to a headless CMS (Contentful at first, eventually DatoCMS) so that editors could easily change copy, graphics, etc. But editing geometries (the shapes and positions of things) still required QGIS knowledge or at least the ability to manually edit geoJSON files.
We launched this as a MVP with the intent of rapidly following up with additional features (blue-dot positioning, turn-by-turn routing, audio directions, a better codebase, etc.) At the end of the project we ended up open-sourcing it (after much begging and pleading with The Powers That Be), but then soon thereafter abandoned it altogether :( Just as well, really, because the code was really terrible -- I can say that with confidence because I wrote it, lol. But it's still an interesting problem space. As far as I know, there isn't a ready-built solution for this sort of stuff... especially not an open-source/source-available one. Some commercial solutions have limited indoor support, but as of our last evaluation (early 2021), none of them were especially powerful, elegant, or user-friendly. Hope that keeps evolving!
-----------
Fast-forward to my current job, and we're now working on a similarly graphical frontend app. I now work for a solar company, and we're designing a map-like tool to help installers plot out the PV modules (the technical term for a single "solar panel") on their customers' roofs, which then enables historical monitoring of various metrics like power output, temperature, etc.
Aside from the graphical challenges (rendering React states to Canvas shapes), there were also some interesting frontend engineering challenges, like how to plot tens of thousands of points in a chart while allowing real-time scrubbing in a timeline, but where no two datapoints were ever at the same time, because the devices we were working with all communicated in a one-at-a-time queue.
Anyway, I won't get too much into the details here... nothing that exciting, just a lot of nuance. What we're building is like a super-simplified version of https://www.opensolar.com/3ddesign (which is awesome! check it out).
---------
But overall, I just love that I get to work on these sorts of fun apps instead of bog-standard blog pages or ecommerce sites. Even a few years ago, things like this w...