We are having atrocious READ/WRITE latency with our PG database (api layer is django rest framework). The table that is the issue consists of multiple JSON BLOB fields, with quite a bit of data— I am convinced these need to be abstracted to their own relational tables. Is this a sound solution? I believe it is the deserialization in these fields of large nested JSON BLOBS that is causing latency. Note: this database architecture was created by a contractor. There is no indexing or relations existing in current schema. Just a single “Videos” table with all metadata stored as Postgres JSON field type blobs.
EDIT: rebuilding the schema from the ground up with 5-6GB of data in the production database (not much, but still at the production level) is a hard sell, but I think it is necessary as we will be scaling enormously very soon.
When I say rebuild, I mean a proper relational table layout with indexing, fk’s, etc.
EDIT2: to further comment on current table architecture, we have 3-4 other tables with minimal fields (3-4 Boolean/Char fields) that are relationally linked back the Videos table with a char field ‘video_id’, that is unique on the Videos table. Again, not a proper foreign key so no indexing.
imo, json dt should be an intermidiary step in db struc in rdb, never the final. Once you know & have stable columes, unravel the json into proper cols with indexing, it should improve the situation
if youre having issues with 5gb, you will face exponential problems when it grows due to lack of indexing
Cheers for the response (and affirmation). After some latency profiling I am convinced proper cols with indexing will vastly improve our situation since the queries themselves are very simple.
Depending on how much of the data in your json payload is required, extract data into their own table/cols. And store the full payload in a file system/cloud storage.
I don't know about PG, but with MariaDB, a nice way to find bottlenecks is to run SHOW FULL PROCESSLIST in a loop and log the output. So you see which queries are actually taking up the most time on the production server.
If you post those queries here, we can probably give tips on how to improve the situation.
Bit hard to tell without some idea of the structure of the data, but my experience has been storing blobs in the database is only a good idea if those objects are completely self contained i.e. entire files.
If you write a small program to check the integrity of your blobs i.e. that the structure of the json didn't change over time, you may be able to infer a relational table schema that isolates those bits that really need to be blobs. Too leave it too long invites long term compatibility issues if somebody changes the structure of your json objects.
I had this issue at a previous job where we would query an API (AWS actually) and store the entire response payload. As we started out we would query into the JSONB fields using the JSON operators, however at some point we started to run into performance issues and ended up "lifting" the data we cared about to columns on the same record that stored the JSON.
Are you just doing primary key lookups? If so, a new index won’t do much as Postgres already has you covered there.
If you have any foreign key columns, add indexes on them. And if you’re doing any joins, make sure the criteria have indexes.
Similarly, if you’re filtering on any of the nested JSON fields, index them directly.
This alone may be sufficient for your perf problems.
If it isn’t, then here’s some tips for the blobs.
The JSON blobs are likely already being stored in TOAST storage, so moving them to a new table might help (e.g. if you’re blindly selecting all the columns on the table) but won’t do much if you actually need to return the JSON with every query.
If you don’t need to index into the JSON, I’d consider storing them in a blob store (like S3). There are trade offs here, such as your API layer will need to read from multiple data sources, but you’ll get some nice scaling benefits here and your DB will just need to store a reference to the blob.
If your JSON blobs have a schema that you control, deprecate the blobs and break them out into explicit tables with explicit types and a proper normalized schema. Once you’ve got a properly normalized schema, you can opt-in to denormalization as needed (leveraging triggers to invalidate and update them, if needed), but I’m betting you won’t need to do any denorm’ing if you have the correct indexes here.
And since you have an API layer, ideally you’ve also already considered a caching layer in front of your DB calls, if you don’t have one yet.
First of all, I think the caching layer (which we currently don’t have) is going to be a necessity in the coming weeks as we scale for an additional project (that will be relying on this architecture)
Second of all, it is just PK lookups. We don’t actually have a single fk (contractor did not set up any relations), which makes me think moving all of this replicated JSON data from fields to tables may help.
The queries that are currently causing issues are not filtering out any data but returning entire records. In ORM terms, it is Video.objects.all(), and from a URL param in our GET to the api, limiting the amount of entries returned. What’s interesting is this latency scales linearly, and at the point we ask for ~50 records we hit the maximum raw memory alloc for PG (1GB) causing the entire app to crash.
The solution you propose for s3 blob store is enormously fascinating. The one thing I’d mention is these JSON fields on the Video table have a defined schema that is replicated for each Video record (this is video/sensor metadata, including stuff like gps coords, temperature, and a lot more).
So retrieving a Video record will retrieve those JSON fields, but not just the values: the entire nested BLOB. And does so for each and every record if we are fetching >1
Would defining this schema with something like Marshmallow/JSON-Schema be a good idea when you mention JSON schemas we control? As well as explicitly migrating those JSON fields to their own tables, replaced with an FK on the Video table?
I do want to emphasize that the S3 approach has a lot of trade offs worth considering. There is something really nice about having all of your data in one place (transactions, backups, indexing, etc... all become trivial), and you lose that with the S3 approach. BUT in a lot of cases, splitting out blobs is fine. Just treat them as immutable, and write them to S3 first before committing your DB transaction to help ensure consistency.
Regarding JSON schema, if you have a Marshmallow schema or similar, yes that’s a wonderful starting point. This should map pretty closely to your DB schema (but may not be 1-to-1, as not every field in your DB will be needed in your API).
I’d suggest avoiding storing JSON at all in the DB unless you’re storing JSON that you don’t control.
For example, if the JSON you’re storing today has a nested object of GPS coords, temperature, etc.. make that an explicit table (or tables) as needed. The benefits are many: indexing the data becomes easier, the data is stored more efficiently, the table will take up less storage, the columns are validated for you, you can choose to return a subset of the data, etc… You will not regret it.
You can index on fields in JSONB, but I don’t believe that’s what the op is solving for here.
In either scenario, I’d still generally encourage avoiding storing JSON(B) unless there isn’t a better alternative. There are a lot of maintenance, size, I/O, and validation disadvantages to using JSON in the DB.
Unrelated to post, but as you seem well informed in the field, would you agree that if a schema is not likely to change and is controlled as you put it, there is no reason to attempt to store that data as denormalized document?
Or at least as you suggest if required for performance the data would still be stored denormalized and where needed materialized / document-ized?
At my current company, there seems to be a belief that everything should be moved to mongo / cosmo (as document store) for performance reasons and moved away from sql sever. But really I think the issue is the code is using an in house orm that requires code generation for schema changes and probably less than ideal performance query generation.
But then I am also aware of the ease of horizontal scaling with the more nosql orientated products, and trying to be aware of my bias as someone who did not write the original code base.
> would you agree that if a schema is not likely to change and is controlled as you put it, there is no reason to attempt to store that data as denormalized document
As a general rule of thumb, yes. Starting with denormalization often opens you up to all sorts of data consistency issues and data anomalies.
> Denormalization is a strategy used on a previously-normalized database to increase performance.
The nice thing about starting with a normalized schema and then materializing denormalized views from it is that you always have a reliable source of truth to fall back on (and you'll appreciate that, on a long enough timeline).
You also tend to get better data validation, reference consistency, type checking, and data compactness with a lot less effort. That is, it comes built into the DB rather than introducing some additional framework or serialization library into your application layer.
I guess it's worth noting that denormalized data and document-oriented data aren't strictly the same, but they tend to be used in similar contexts with similar patterns and trade-offs (you could, however, have normalized data stored as documents).
Typically I suggest you start by caching your API responses. Possibly breaking up one API response into multiple cache entries, along what would be document boundaries. Denormalized documents are, in a certain lens, basically cache entries with an infinite TTL... so it's good to just start by thinking of it as a cache. And if you give them a TTL, then at least when you get inconsistencies, or need to make a massive migration, you just have to wait a little bit and the data corrects itself for "free".
Also, there are really great horizontally scalable caching solutions out there and they have very simple interfaces.
Thanks for your response. The comparison between infinite ttl cache entries and a denormalized doc is an insight I can't say I've had before and makes intuitive sense
This is an affirmation I’ve been longing to here, lol!
I’ve already done the legwork, cloning to the current prod DB locally and playing around with migrations, but the fear of applying anything potentially-production breaking is scary to a dev who has never had to work on a “critical” production system!
I would recommend setting up a staging app with a copy of the production database, testing a migration script there, then running the same script on production once you're confident.
1. What are the CRUD patterns to the "blobby data".
2. What are the read patterns and how much data needs to be read.
Until Read/Write are properly understood the following solutions should be considered as general guide lines only.
If staying in PG:
JSON can be indexed in Postgres
You could also support a hybrid JSON/Relational model giving the best of worlds.
Read:
Create views into the JSON schema that model your READ access patterns and expose them as IMMUTABLE Relational entities. (Clearly they should be as light weight as possible)
Modify:
You can split the JSON blobs into their own skinny tables. This should keep your current semantics and facilitate faster targeted updating.
Big blobby resources such as video/audio should be managed as resources and not junk up your DB
Warning:
Abstracting the model into multiple tables may cause its own issues depending on how you ORM map your entities.
Outside the Box Thinking:
-Extract and transform the data for optimized reading.
-Move to MongoDB or a Key Value store
Conclusion:
What are the update patterns?
-is only 1 field being updated
-inter-dependencies of the data being updated
How are "update anomalies" minimized
You will need to create a migration strategy to a more optimal solution and would do well to start abstracting with views. As the data model is improved this will be a continuous process and the data model can be "optimized" without disturbing the core infrastructure requiring rewrites.
I had to move a MongoDB to PG database on my new job (old contractor created MVP, I was hired to be the "CTO" of this new startup) and I had some problems at first, but after I created the related models and added indexes, everything worked fine.
As someone said, indexes are the best to do lookups. Remember your DB engine do lookups internally even if you are not aware (joins, for example), so add indexes to join fields.
Another thing that worked me for me (and I dont know if it's your case), was to add trigram text indexes, which make it faster to do a full text search. Remember, anyway, that adding a index makes search faster, but insert slower, so be careful if you are inserting a lot of data.
I don't think the latency issues are necessarily related to the poor schema. I'd say to dig into the query planning for your current queries and figure out what's actually slow, since it may not be what you expect.
Rearchitecting the schema might be worth doing. From the technical side, PG is pretty nice about doing transactional schema changes. I'd be more worried about the data though. Are you sure that every single row's Json columns have the keys and value types that you expect? Usually in this type of database, some records will be weird in unexpected ways. You'll need to find and account for them all before you can migrate over to a stricter schema. And do any of them have extra unexpected data?
-Learn the json in-build functions and see if one of them can replace one you made ad-hoc
- Seriously, replace json with normal tables for the most common stuff. That alone will speed up things massively. Maybe keep the old json around in case, but remove when it become old(?)
- Use views. Views allow to abstract over your databaee and allow to change internals
- If a big things is searching and that searching is nkind of complex/flexible, add FTS with proper indexing to your json then use it as first filter layer:
SELECT .. FROM table WHERE id IN (SELECT .. FROM search_table WHERE FTS_query) AND ...other filters
This speeeeeedupppp beautifully! (I get sub-second queries!)
- If your query do heavy calculations and your query planner show it, consider move them into a trigger and write the solved result into a table, and query that table instead. I need to loans calculations that requiere sub-second answers and this is how I solve it.
And for your query planner investigation, this handy tool is great:
Large blobs are not the use case of relational databases - this is the starting point for any such discussion. I have 2 current projects where I am convincing the app builders (external companies, industry-wide used apps) to change this, keep relational data in the database and take out the blobs, so far is going better than expected.
I'm not sure if this is in the spirit of the thread but I've been working on a way to allow reviews of gameplay in video games. In short, you upload a video of you playing the game and someone who's an expert can review it.
I currently have a UI with the comments down the side of the screen which looks like this:
However the downsides of this is that you can't see all the comments at once. I'm not a UI/UX designer AT ALL so I'd really appreciate some pointers around how to think about making this better! The original post mentions "close to solving", I think I am pretty close but it's still not quite right and while I'm not out of ideas yet, I'd appreciate feedback if solving this is obvious to someone else.
So maybe the problem with showing all the comments at once is that there are too many, and when showing one at a time they are not shown for long enough.
How about breaking the play into chapters/zones/rooms/segments (whatever makes sense for the game) then showing all the comments for that segment. Once the segment ends, there would be a replay button if they missed anything on the first play while reading comments, and a next segment button to carry on.
Interesting time spans could be marked for slow motion, boring bits played at double time.
There would be high level navigation between segments with thumbnails and comment counts. Buttons to skip between “pivotal moments”, maybe with voting to highlight them.
It's totally crazy but can be made much more coherent. Would be useful to have comments at a specific time and place on the screen just for very accurate pointers/comments.
I find that my normal model for reading comments on videos across platforms is to not read them much of the time, but if it's a really interesting video go look at the comments and it's ok if the video is for example, fully minimized or off screen etc while I read them.
I don't know how normal my use is though or if that's at all helpful.
The default should be to show one comment at a time, because that's convenient and quick to get into, but also with an option (maybe just a scroll down) to view all comments. One, that helps the reviewer get an overall idea of what kind of things the submitter is looking for, if they want that, and two, some submitters are inevitably gonna screw up, posting at the wrong times or asking overall summary questions that should be asked at the end right at the beginning or somesuch. So an All Questions button or similar should be there as an escape hatch, but not the primary UI.
We are working on a totally new way to do cold fusion, our only problem is getting enough new fuel into the reactor without disturbing the running process.
That actually does make for a fairly efficient (better than fusion in energy per unit fuel mass) reactor design in principle, but you need a sub-solar-mass black hole (in the 1 billion - 100 billion ton range), and there's no known practical way to produce one.
Could you use tiny pellets fed in by a linear actuator and gravity, rotary loader, or some kind of a conveyor belt? If you want an off-the-shelf solution that's easy to reload, perhaps you could repurpose the loading mechanism of a machine gun to dispense the pellets.
I'm working on a prototype that uses the compositional game theory [1] and adapts it to be able to reliably predict the order complexity of functors and their differences between states.
A huge bonus there would be when the order difference can be represented in a graph, so that tesselation or other approaches like a hypercube representation can be used for quick estimations. (that's what I'm aiming for right now)
If successful, the next step would be to integrate it into my web browser so that I can try out whether the equilibrium works as expected on some niche topics or forums.
By no means am I an expert, but just another parent on HN.
1. Ritualize
I notice a pattern with my kids: going to bed is a ritual, and any deviation is reason enough for them to leave the bed.
2. Slow down in advance
Going to bed after playtime is impossible. So cut screens, playtime, play/listen some quiet music, read books or newspaper (again, no tablet/reader), at
least 1h before bed time.
3. Recap the day
Remind your kids of their day, activities and make them aware of the fatigue. Works better when in bed, with mine.
4. Stay with them if they're afraid
Learn why they're afraid, teach them why there's no reason to be afraid. I've had to hang a sock to the door every night for months to scare tigers away :D
It just works ^^
Every parent knows the pain and every kid has their own back story and the relationship with the parent(s) is key to finding a way into bed.
Eventually, they will sleep.
In our case, we settled that unless exceptional situations, our kids had to fall asleep in their own bed, because we wanted/needed our intimacy. To get that, I had to stay in my kids' room for as long as 2hrs for months, but didn't let go. Today, going to bed is thankfully not a situation anymore.
I'm not sure this is in any way helpful, but here's my shared experience and learnings. YMMV.
in general aim for repetition and stringing activities together.
So for us, when I first started to do this. Each night they get a 'treat' but to get that treat they first need to be ready for bed - eg bedroom ready to sleep in, correctly dressed/ washed etc..
then after the treat they must choose a calming activity - ideally in their bedroom eg reading (nothing that gets their heart rate up) for 30-60mins then they must bush their teeth
It's this point we say time for bed, but we allow them to carry on reading for another 30-60min then it's lights off
if they don't do the activities/ actions after the treat, then we warn them that they'll not get one tomorrow etc. (and really do what you say)
also you may need to flexible on the activities until they get into the swing of it
Apparently I also still have a sense of humor. Or maybe I don't, because perhaps pretending one doesn't get the joke of doubling down on xkcd silliness is perhaps a joke in itself, which I didn't get.
Benadryl, no. But melatonin, sometimes, yes. My rule is to have a hard cutoff time after which it's better to take melatonin than to continue the cycle of whining, sleep deprivation, and next-day misery. The cutoff is late enough to have plenty of time to try all the other things involving wind-down rituals. It is not an every day thing. I found that having the consistency actually helps establish the rituals too.
Also, bedtime trouble usually means not enough outside time and physical activity during the day. Or that the kids want more of the parent's time.
Look into the research behind melatonin use. I’m not a doctor and certainly long term use of diphenhydramine is associated with neurological problems in old age, but I’m not sure melatonin should be used as a simple hypnotic as you are suggesting. It’s natural but so is testosterone. Hormones may not be good to tinker with. I say that as a long time user of melatonin. At the very least you may want to stick with lower dosages- nothing over 1mg which is about as low as is easy to find.
Kids versions don't come in anything higher than 1mg and one can make it a half-dose quite easily. It's really more of a last resort thing, and definitely not for every night's bedtime. How last resort? Maybe once or twice a month. Now that my kids are a bit older and bedtime rituals are established it's even more rare.
I realize that some parents reach for it every night and this is not something I'm suggesting.
For mine: A reading light with a remote-control timer. We read a story together, brush teeth, then they get 10 minutes of independent time with their own storybook. But kids vary, so good luck.
Best trick I've learned (and I have more than twice as many kids as the average citizen here :-) is to:
1. make sure they don't fall asleep with something they cannot keep all night (i.e. while you are singing to them, rocking them, sitting next to them or when they are drinking a bottle of milk etc)
2. make sure they understand that even if you leave the room it is just temporarily. Small kids are - for good reasons - very afraid of being forgotten or left alone.
2.1 Using a timer to remember to visit the room regularly and often as they learn to sleep alone can help a lot
2.2. Increase the interval each day. I increased it by two minutes each day.
2.3 If the kids are happy in their bed, continue to visit their room at the scheduled time: you don't want them to think that you forget them if they don't cry.
Using this method I've got my last few kids to enjoy going to bed and sleep better in less than a week fo each of them.
Looks pretty slick. I am not a gamer, and the controls feel very backwards to me. You need to hold the mouse button to look in different directions, which makes it feel like you're dragging, but the direction you move it isn't the direction you're dragging the view. I don't think mouse capture is a good idea in the browser, and if you're aiming at museums and galleries, maybe reversing the mouse direction to make it like familiar dragging would be better.
Edit: There's a bug where if you start dragging with the mouse and let go the mouse button outside the 3D view, it acts like the button is still held down (a bit like mouse capture) which was easier but quite confusing.
It seems unusable with a touch screen on desktop.
The fact that you can fly is useful but non-obvious. I ended up down at floor level and wondering how to see the pictures on the walls.
The demo is in a /fr/ path, but is in English (Chrome offers to translate it to English, because it somehow thinks the English words are French), but then some parts of the interface like "Share your place" are in French.
I reiterate the sibling comment about mouse control, either use the actual mouse locking APIs of browsers, or make drag feel like drag.
Additionally, I think it would be best if by default you were stuck to standing head height, then you can either provide buttons to actually move up or down, or lean more into the game aspect and allow the user to jump. Right now it feels like you are floating around with a little drone or something.
In the vein of controls, please please please support WASD too. I understand if you instruct with arrow keys, since for non-gamer users it might be more obvious, but support WASD (or equivalent of what WASD is in QWERTY keyboards) anyways, for 2 reasons: it is much more ergonomic for people who use a mouse on their right hand (I myself am left-handed but use the right hand for mouse anyways), and is more ergonomic for some laptop users, since many laptops have half-size vertical arrow keys which are uncomfortable to press all but momentarily.
2. When viewed full-screen on a 4k monitor, textures are too low resolution. A handwritten note on the wall is unreadable.
3. Lighting is too simple. Because that’s not an FPS shooter you probably don’t need dynamic lightning nor day/night cycle, but it’s still hard. Ideally you need these multiple PBR textures everywhere, and correspondingly complicated pixel shaders.
2 – can you possibly replace them with higher resolution ones after the scene is already running? Ideally, gradually with a blend over ~1 second.
3 — I see. Still, you could pre-compute local illumination automatically in the editor, and bake it somewhere. Maybe into vertex attributes, maybe into another lower-resolution R8_UNORM set of textures.
2- --> The person Who builds the space in 3D With free-visit 'builder' decides how much he squeeze down the texture quality.
His choice depends on the target: Smartphone (low quality), computer with big screen (high quality texture)
3- --> I will see with client feedback. I do not want at this point to over-engeneer free-visit.
First I must find my market.
Frustrated by the degree of manual programming process in production metal machining. The industry exists largely on inertia. I would like to resolve this by applying standard optimization algorithms to a set of known machining strategies plus machine, work-holding, material, part and tool inputs. Have already analyzed the problem space to some extent and will be touring a huge production facility next week to better understand best-in-class processes from large established players. Need someone to either wrap existing simulation algorithms (any CAM system) or write enough of one (not that hard, the solution space is extremely multivariate but well understood and well documented) to make it feasible (not too hard for 2.5D machining). You can get as intellectual as you like in the solution, but remember perfect is the enemy of done. Value is huge, happy to split equity on a new entity to resolve if a workable solution for the easier subset of parts emerges in the next few weeks.
We run about 40-50 CNC. Lots of our engineering time goes in to planning, on how to step by step machine a component so that it can reach mentioned tolerances. Sometimes required tolerance are at or below the machine accuracy. Are you going to solve this also ?
This may already be solved, but one of the last pieces remaining in my quest to be Google-free is an interoperable way to sync map bookmarks (and routes, etc) between different open source mapping apps. I can manually import/export kmz files from Organic Maps and OsmAnd, and store them in a directory synced between different devices with Nextcloud, but there's no automatic way to keep them updated in the apps, and so far I haven't found a great desktop app for managing them either. The holy grail would be to also have them sync in the background to my Garmin Fenix, but I am not aware of a way to sync POIs to a Garmin watch in the background.
Related: I'd love to have an Android app with a shortcut that allows me to quickly translate Google Maps links into coordinates, OSM links or other map links. There is a browser extension that does this on desktop, so if anyone is looking for a low hanging fruit idea for an Android app, this might be a fun idea (if I don't get around to it first).
I'm using Nextcloud to host my calendar. On my work Mac, I connect to it using Fantastical. On my personal Ubuntu machine I use GNOME Calendar, and on Android I use https://github.com/Etar-Group/Etar-Calendar
Everything is seamless for me, though admittedly I'm not a super heavy calendar user.
I plan to do a write up on my whole Google-free setup, but I haven't done it yet, unfortunately.
I got a VPS and installed Nextcloud with Docker. I would self-host on my own server, but I'm too nomadic for that at the moment. I think the /e/ foundation has a decent managed Nextcloud setup.
We are experiencing very high CPU load caused by tinc [0], which we use to ensure all communication between cloud VMs is encrypted. This is primarily affecting the highest traffic VMs, including the one hosting the master DB.
I am starting to consider alternative tools such us wireguard to reduce load, but I am concerned of adding too much complexity. Tinc's mesh network makes setup and maintenance easy. The wireguard ecosystem seems to be growing very quickly, and it's possible to find tools that aim to simplify its deployment, but it's hard to see which of these tools are here to stay, and which will be replaced in a few months.
What is the best practice, in 2021, to ensure all communication between cloud VMs (even in a private network) is encrypted?
Apart from some smaller projects building on top of WireGuard, there's Tailscale [1]. One of the founders is Brad Fitzpatrick who worked on the Go team at Google before and built memcached and perkeep in the past.
Outside of the WireGuard ecosystem there's ZeroTier [2] which has been around for a while and they're working on a new version; and Nebula [3] from Slack, which is likely to be maintained as long as Slack uses it.
There might be others, but with tinc these four are the ones I've seen referred to most often.
+1 for Tailscale, the product is great. I've used it in a very limited scale but can vouch for quality and performance. No CPU issues at all (even on rPi).
Similar to Tailscale is the Innernet project, which has similar goals but is fully open source (also built on Wireguard). I've heard that set-up is a bit more painful, but for those who are interested in FOSS or self-hosting, it might be worth looking into.
I’m facing an issue where I store small binary data blobs within a Postgres column in order to benefit from delete cascades.
I’m considering moving the binary data into S3 and then doing the sync layer on the server (which means the front end requests the data from the backend and is given it back as a JSON object with base64 values).
Doing this manually via code isn’t impossible, just API intensive, so I’m wondering if this is a solved issue for anyone.
The why: The JSON blobs are recordings of words and sentences that can be copied between articles.
I try to find an agile project management tool that works for us. We run on what many would call Scrum (it’s not actually Scrum).
We are on JIRA now, and it’s … JIRA. We tried basically any other tool, including Excel (yes, that is somewhat possible).
My problem generally is that tools are slow, planning is cumbersome, visibility is limited and reporting for clients is often even more limited.
Heck, I’d even write my own tool if I knew it would help others, but I am concerned it’s too close to what we already have for anyone to actually migrate.
I've recently started using ClickUp for managing my helpdesk and development work and I like it a lot. I don't do scrum myself but the product claims to be useful for that kind of work, as well as many other approaches and use cases.
We use Restyaboard for Agile marketing in that you will be able to manage all your projects, teams, and clients from one single space.
https://restya.com/board/demo
1. I want to create way to generate electrical power without pollution. Basically, a closed cycle process that releases no pollutants, or electronic waste.
2. I want to do everything I can to eliminate gender bias in the world.
Regarding 1, if you can accept some pollution at the beginning, hydro-electrical can be a solution, albeit probably not for a global scale.
We have a small hydro-electrical plant una River near my house and really it's no big deal, it fits very nicely in the surrounding environment and it produces clean energy.
It's also educational because since the river is near the city small children classes can visit it and learn about it.
Regarding hydro, you can go micro to power a house or some small comunity. There are lot's of books on microhydropower, but get a look at this fantastic post at ludens.cl
I love hydroelectric power and I want the circuits I design for solar to be capable of utilizing the raw power from a small turbine as well without any hardware modifications.
Rather than generating electricity directly, it might be more practical to reduce electricity consumption using other approaches:
Geothermal can be a solution for generating electricity directly, but if you'd like to minimize electronic waste perhaps it would be easier to use it to replace alternative energy sources for HVAC purposes.
Biofuels (eg: plant bamboo, grow it, then burn it) can also technically be closed cycle energy sources.
Solar water heaters can also reduce electrical or fossil-fuel-based energy consumed for generating hot water.
I worked for years studying how to use microcontrollers and after a lot of determination now I have a $750,000 grant to build solar systems that are fireproof so you can install them anywhere. It will be a few years of work to get something suitable made but I have full confidence it can be done.
I am also spending much time lately in the SF kink community to build a fundamental understanding of the biases people have experienced in life with respect to their gender identity, and am strongly considering HRT so I can live life on the other side and experience the prejudice first hand.
The API docs' example URLs can't be copied and pasted, as "%20" gets inserted. I'm probably not the target audience for the API, but it's a neat idea. I think people will end up with images described with tons of magic numbers that relate to each other in invisible ways and become unmodifiable and unmaintainable. Variables might help, as might relative positioning and sizing of elements.
Some feedback on the Designer:
The size setting dropdown is quite strange. Choices of unfamiliar destinations don't seem to make sense, and some of the things that do seem familiar come out an unexpected size and shape (e.g. "infographic"). The pixel sizes are clearer, but better would be handles on the canvas that can be dragged. There's also a typo: "Choose form a list of sizes".
Circles don't get resized? I can drag handles to make the apparent bounding box bigger, but the circle doesn't change: https://imgur.com/a/KhYluKj. Other shapes seem okay. Chrome 89 on Linux.
The tutorial walkthrough pops up every time I go to the designer, even though I've been right through it.
Seems like it would be a lot easier and a lot more powerful to use SVG instead of a giant string of URL parameters.
Your service could provide pre-made templates and an editor, and expose textfields, images, fonts, etc options via URL parameters. Then your service just has to render the SVG and return it as an image with the requested dimensions/format.
I have posted this here before- hexafarms.com. I am trying to use ML to discover optimal phenotype for growing plants in vertical indoor farms to a. have the higest quality produce b. to lower the cost of producing leafy green/med plants, etc. within cities itself.
Basically, every leafy green (and herbs, and even mushrooms), can grow in a range of climatic condition (phenotype, roughly) ie temperature, humidity, water, CO2 level, pH, light (spectrum, duration and intensity) etc. As you might have seen around the world there is a rise in indoor vertical farms, but the truth is that 50% of those are not even profitable. My startup wants to discover the optimal parameters for each plant grown in our indoor vertical farm and eventually I would let our AI system control everything (something like alphaGo, but for growing plant X (lettuce, kale, chard, ). Think of it as reinforcement learning with live plants! I am betting on the fact that our startup will discover the 'plant recipes' and figure out the optimal parameters for the produce that we would grow. Then, the goal is that cities can grow food cheaper in more secure and sustainable way than our 'outsourced' approach in country side or far away lands.
So now I have secured some funding to be able to start working on optimizations, but I realized that *hardware* startups are such a different kind of beast (I am a good software product dev though, I think). Honestly, if anyone with experience in hardware related startups (or experience in the kind of venture I am in) would just want to meet me and advise me, I would take it any day. Being the star of the show, it's hard for me to handle market segmentation, tech dev, team, next round of funding, European tech landscape, etc. I am foreseeing so many ways that our decisions can kill my startup, all I need is advise from someone qualified/experienced enough. My email: david[at]hexafarms.com
Sounds similar to what I read a long time ago about a big tomato farm in the Netherlands... Have you tried talking to actual farmers of that produce? Universities? Agricultural faculties do a lot of research in that direction.
Expensive, quickly perishable produce might be able to compete, otherwise I guess free water and energy from above in the "remote" classical farming will be hard to beat.
And then my naive guess would be that to generate enough data for a "ml" approach not only by name might be somewhat expensive.
This sounds so negative, but this is not my intention... I wish you all the best and hopefully will stumble upon a success story in the future :-)
This isn’t an answer to your ML question, but it is an answer to your problem.
I heard about a greenhouse company that has programmed their climate control to match “best growing conditions historical weather”. So, they ask local experts what year / location had the best X and then they use that region’s historical weather and replay it in their greenhouse. I thought that was brilliant!
(Just realized this was Kimbal Musk that mentioned this)
When I studied farming back in 1998-1999 we once visited a greenhouse and one interesting thing I picked up was that by observation some gardeners had realized that lowering the temperature a bit extra an hour or two before sunrise they could get their flowers to be more compact instead of stretching.
This had replaced shortening hormones in modern gardening (or at least at that greenhouse, but my understanding they were just doing the same thing as everyone else).
I guess there is a lot more to learn for those who have scale enough to experiment and patience to follow through.
Reminder to focus on nutritive content, flavour, and crop diversity, not just yield. The past 100 years of industrial scale agriculture, with the singular goal of maximizing yields, has done incredible harm. (This has come up on HN repeatedly, so I trust you've seen it, but it's worth championing)
I agree that micronutrient content has decreased in the past century. Some might be because of scale, some might be that yield gains are mostly driven by macronutrients and water, not micronutrients, it could be selecting varieties that taste better, or it could be depleting the soil.
That said, the US has an obesity epidemic, so there's no shortage of macronutrients. Macronutrient shortages also seem rare. Scurvy and rickets aren't exactly problems.
There's some great research on using evolutionary computation to explore plant growing recipes (light strength, how long to leave the lights on, etc). In one experiment, researchers discovered that basil doesn't need to sleep - it grows best with 24 hours of light per day. Risto Miikkulainen shared the experiment on Lex Fridman's podcast: https://youtu.be/CY_LEa9xQtg?t=27m7s I believe this is the paper describing that experiment: https://journals.plos.org/plosone/article?id=10.1371/journal...
This sort of ml problem is characterized by relatively expensive data labeling. Hence, hiring an expert or mixture of experts, and modeling the crop responses to their choices, will save you a lot of hill climbing The wrong part of the decision space
I know this isn't going to sound as sexy as AlphaGo for plants, but I really think this is a classic multilinear optimization problem once you've properly labeled the data and defined the dynamics between the plants / other organisms (e.g., aquaponics). You're looking to optimize multiple variables across a set of known constraints and I think if you properly defined these constraints you could save a lot of headache / buildout by leveraging a pre-exsting toolset like Excel with the Excel Solver add-in an a couple hundred user defined functions. We're talking 1% of the work to get something useable and product-market-fitable with automatic output of graphs, etc, that clients could tune and play with locally without you needing to actually share the source sauce. Eventually you could switch to Python for something more dynamic / web based.
How visually implement process in the app? I.e. how to guide users over complex process they need to do in the app to achieve success?
The process might span different medium (write email, do something in the app, check twitter, etc) and different activities multiple days. How to make sure they know what they should do next? Checklist? Emails? Slack? Wizard?
Interesting... My to-go solution for this would be a detailed wiki page with screenshots, and link to that from a bunch of places. But I guess that's not an ideal solution really.
This would probably require a lot of front loaded work in your case, but if you need to train a boat load of people with very few (or zero) trainers, my favorite way to do it is (Atlassian’s Atlaskit Onboarding/Spotlight components)[https://atlaskit.atlassian.com/packages/design-system/onboar...]
I am a bit ashamed to ask about such a trivial topic on HN, but I am not really sure how AJAX in Wordpress plugins works.
I have a plugin that exports some WooCommerce orders into XLS. I would like to add a progress bar via AJAX, because the export may take very long for thousands of orders. But I am not really sure how to use AJAX in context of Wordpress specifically.
I would love to see a minimal functional example, a simple plugin that does something similar. So far, all the plugins I saw were pretty convoluted and I lost my track around the code.
(On a related note: a library of elementary examples for Wordpress plugin development would be nice. Like "This is how you create a menu entry.")
In my experience, accurately reflecting the progress of and AJAX request, so I’ve seen a lot of people (myself included) take the lazy way out and just show an indeterminate spinner or bar just to show that stuff is happening.
The idea entered my mind, but I am not happy with such cop-out, especially on my own site ... I would at the very last like to see iteration progress, which, while not 1:1 with time, is at least informative.
It takes a bit of code but if you know the length of the response you're expecting then you can use XHR's "progress" event. Just be aware that the event will happen frequently and contain all the data so far, to avoid inefficient parsing and substring related memory leaks. I think his problem might be more about using JS in WordPress though.
First of all, you'll need to make PHP to display "progress", probably you'll need to override ob_start() or something like that, and find a format that let you append new progress on the response on the fly.
I guess you already have an URL on your Wordpress setup that triggers this export. Let's call it {url}/export.
Wordpress already has jQuery by default included. So'll you need to call that URL using jQuery $.post and then, accordingly to the response, update your progress bar.
There is nothing specifically about Wordpress on this, besides the fact that you need to setup your own URL on Wordpress to do this, and then include your own JS after jQuery. That's all.
If you find this too-complicated, a quick-hack is to create a page on WP Admin called Export Tool, and then on your theme create page-export-tool.php. That .php will be called when visited that Export Tool page.
Ok, I'm double dipping. Another problem I'm trying to solve is: I've got a database of several hundred interesting conversation questions that I've collected over the years. Essentially just strings, though I've attempted to categorize them, rank them, and add other metadata. I'd like to figure out a way to sort them or dedupe them based on semantic similarity, but I'm not sure how to determine semantic similarity without painstakingly going through and manually looking for similar questions. Any suggestions on how to solve this would be welcome.
Put the questions in some semantic embedding space. Now you’ll have a vector representing each question. Then for each question, you can sort all the questions by how far the Euclidean distance is between their vectors. Or use some clustering algorithm like k-means to find clusters.
Yep, exactly this. Check out sentence-transformers https://pypi.org/project/sentence-transformers/0.3.0/, they have some great pre-trained models. Once you have the embeddings you can just compute the cosine similarity.
I am not there yet, but I am trying to make education loan-free and based on equity. The details of the project are written here: https://loan-free-ed.neocities.org
> when that student starts generating income then a small percentage from that income is auto-deducted and distributed to everyone who was involved in that student's education.
Your idea, if implemented well, may end up being a net positive for society, but I can't help imagining a future where every child, from the moment they are born, has a biometric ID connecting them to a consortium of companies which provide their education, health care, housing, energy, internet connectivity, transport, media access, and so on.
It would be like living in a company town, being paid in company scrip, except you wouldn't notice the restrictions (as long as you kept earning). If you ever increased your income, your consortium might let you choose whether you want to upgrade your housing or your health care plan, but if you lost your job, they'd force you to take one of their choosing and downgrade your plans if it had a lower salary.
In this dystopia, all consumables from food to toilet paper would presumably be sold by Amazon, and other items like furniture and electronics would be provided as a service so that you rent them from your consortium. The only question is why people wouldn't try to undo this system through the political process, but then we might ask that about the current system.
> Your idea, if implemented well, may end up being a net positive for society
Thank you!
> but I can't help imagining a future where every child, from the moment they are born, has a biometric ID connecting them to a consortium of companies which provide their education, health care, housing, energy, internet connectivity, transport, media access, and so on.
My idea will not have that side-effect because not only is the project non-profit and open source, but it is also decentralised. And if we keep thinkng of a dystopian future then we won't be able to do anything positive unless we become some sort of social revolutionaries. I don't have those skills. But I can think of small ideas to make a positive impact that benefits everyone though. The idea I listed in my original post above is very simple, it helps teachers, lecturers, or anyone who contibutes to a persons monetary decent life via education get appropriately paid for their efforts, and everyone(i.e. businesses) who benefits from an educated person should contribute towards that.
It is a simple idea but notoriously diffcult to deploy because there is a possibility of this getting caught up in a political slugfest.
Working on trying trying to extract the most benefit from my suicide while reducing the risk of of damage. Right now there's two issues:
* I'd like to donate my kidneys, liver, and lungs as a living donor. However I've heard that there may be a psychological screening component for being a living donor which may detect that I'm planing on killing myself and trigger intervention. Is there a way around it? It's not critical for success but it would be nice to give back to someone else a chance at life, but I'm not sure if it's doable.
* I want to minimize the possibility of being discovered and identified after death. I've selected a heavily wooded site that's time consuming to access on foot and selected timing to be early autumn some years from now once I've acclimated my acquaintances to my absence. The concern however comes from the fact that I have a security clearance and my fingerprints are in the system. I'm trying to figure out if it's a real concern that my body is discovered with fingers intact, and what needs to be done to mitigate it. Or if seclusion, exposure, and scavengers are enough to cover that possibility and it's not a real concern.
I'm looking for a way to integrate a React app with an existing Vue... thing. Don't really need any communication between the two, just displaying it would be fine. My issue is: the Vue code just throws in <script> tags in the html and expects global variables (location instead of window.location), while the React code uses ES6 imports. The only partly working way I found is including <script> tags with a useEffect, but that doesn't play nicely with apex-charts for some reason, and includes forcing the html with a dangerouslySetInnerHTML after importing the existing file as a long string. Sub par, obviously. In addition, I'll probably need to include different vue apps in a couple different instances. Any suggestions? I think I might just keep them separate and open a new tab for the explanatory session. thanks!
Reasoning: helping with the code behind a paper on explanatory AI systems.
Have you tried SingleSPA? I’ve used it in the past to get React apps inside an old AngularJS app to replace parts of it over time and it worked pretty well. Docs say it works with Vue as well but I don’t have any direct experience with Vue at all far less for this kind of task so I’m not really sure it will work but it is worth looking at https://single-spa.js.org/
Have you tried using `useRef` and attaching the Vue component to a parent React node? I haven’t does this with Vue, but have with a vanilla JS chart library.
485 comments
[ 5.0 ms ] story [ 695 ms ] threadEDIT2: to further comment on current table architecture, we have 3-4 other tables with minimal fields (3-4 Boolean/Char fields) that are relationally linked back the Videos table with a char field ‘video_id’, that is unique on the Videos table. Again, not a proper foreign key so no indexing.
if youre having issues with 5gb, you will face exponential problems when it grows due to lack of indexing
If you post those queries here, we can probably give tips on how to improve the situation.
Tangentially related for those who have experience, I am using Django-silk for latency profiling.
If you write a small program to check the integrity of your blobs i.e. that the structure of the json didn't change over time, you may be able to infer a relational table schema that isolates those bits that really need to be blobs. Too leave it too long invites long term compatibility issues if somebody changes the structure of your json objects.
If you have any foreign key columns, add indexes on them. And if you’re doing any joins, make sure the criteria have indexes.
Similarly, if you’re filtering on any of the nested JSON fields, index them directly.
This alone may be sufficient for your perf problems.
If it isn’t, then here’s some tips for the blobs.
The JSON blobs are likely already being stored in TOAST storage, so moving them to a new table might help (e.g. if you’re blindly selecting all the columns on the table) but won’t do much if you actually need to return the JSON with every query.
If you don’t need to index into the JSON, I’d consider storing them in a blob store (like S3). There are trade offs here, such as your API layer will need to read from multiple data sources, but you’ll get some nice scaling benefits here and your DB will just need to store a reference to the blob.
If your JSON blobs have a schema that you control, deprecate the blobs and break them out into explicit tables with explicit types and a proper normalized schema. Once you’ve got a properly normalized schema, you can opt-in to denormalization as needed (leveraging triggers to invalidate and update them, if needed), but I’m betting you won’t need to do any denorm’ing if you have the correct indexes here.
And since you have an API layer, ideally you’ve also already considered a caching layer in front of your DB calls, if you don’t have one yet.
First of all, I think the caching layer (which we currently don’t have) is going to be a necessity in the coming weeks as we scale for an additional project (that will be relying on this architecture)
Second of all, it is just PK lookups. We don’t actually have a single fk (contractor did not set up any relations), which makes me think moving all of this replicated JSON data from fields to tables may help.
The queries that are currently causing issues are not filtering out any data but returning entire records. In ORM terms, it is Video.objects.all(), and from a URL param in our GET to the api, limiting the amount of entries returned. What’s interesting is this latency scales linearly, and at the point we ask for ~50 records we hit the maximum raw memory alloc for PG (1GB) causing the entire app to crash.
The solution you propose for s3 blob store is enormously fascinating. The one thing I’d mention is these JSON fields on the Video table have a defined schema that is replicated for each Video record (this is video/sensor metadata, including stuff like gps coords, temperature, and a lot more).
So retrieving a Video record will retrieve those JSON fields, but not just the values: the entire nested BLOB. And does so for each and every record if we are fetching >1
Would defining this schema with something like Marshmallow/JSON-Schema be a good idea when you mention JSON schemas we control? As well as explicitly migrating those JSON fields to their own tables, replaced with an FK on the Video table?
Regarding JSON schema, if you have a Marshmallow schema or similar, yes that’s a wonderful starting point. This should map pretty closely to your DB schema (but may not be 1-to-1, as not every field in your DB will be needed in your API).
I’d suggest avoiding storing JSON at all in the DB unless you’re storing JSON that you don’t control.
For example, if the JSON you’re storing today has a nested object of GPS coords, temperature, etc.. make that an explicit table (or tables) as needed. The benefits are many: indexing the data becomes easier, the data is stored more efficiently, the table will take up less storage, the columns are validated for you, you can choose to return a subset of the data, etc… You will not regret it.
In either scenario, I’d still generally encourage avoiding storing JSON(B) unless there isn’t a better alternative. There are a lot of maintenance, size, I/O, and validation disadvantages to using JSON in the DB.
Or at least as you suggest if required for performance the data would still be stored denormalized and where needed materialized / document-ized?
At my current company, there seems to be a belief that everything should be moved to mongo / cosmo (as document store) for performance reasons and moved away from sql sever. But really I think the issue is the code is using an in house orm that requires code generation for schema changes and probably less than ideal performance query generation.
But then I am also aware of the ease of horizontal scaling with the more nosql orientated products, and trying to be aware of my bias as someone who did not write the original code base.
As a general rule of thumb, yes. Starting with denormalization often opens you up to all sorts of data consistency issues and data anomalies.
I like how the first sentence of the Wikipedia page on denormalization frames it (https://en.wikipedia.org/wiki/Denormalization):
> Denormalization is a strategy used on a previously-normalized database to increase performance.
The nice thing about starting with a normalized schema and then materializing denormalized views from it is that you always have a reliable source of truth to fall back on (and you'll appreciate that, on a long enough timeline).
You also tend to get better data validation, reference consistency, type checking, and data compactness with a lot less effort. That is, it comes built into the DB rather than introducing some additional framework or serialization library into your application layer.
I guess it's worth noting that denormalized data and document-oriented data aren't strictly the same, but they tend to be used in similar contexts with similar patterns and trade-offs (you could, however, have normalized data stored as documents).
Typically I suggest you start by caching your API responses. Possibly breaking up one API response into multiple cache entries, along what would be document boundaries. Denormalized documents are, in a certain lens, basically cache entries with an infinite TTL... so it's good to just start by thinking of it as a cache. And if you give them a TTL, then at least when you get inconsistencies, or need to make a massive migration, you just have to wait a little bit and the data corrects itself for "free".
Also, there are really great horizontally scalable caching solutions out there and they have very simple interfaces.
I’ve already done the legwork, cloning to the current prod DB locally and playing around with migrations, but the fear of applying anything potentially-production breaking is scary to a dev who has never had to work on a “critical” production system!
1. What are the CRUD patterns to the "blobby data". 2. What are the read patterns and how much data needs to be read.
Until Read/Write are properly understood the following solutions should be considered as general guide lines only.
If staying in PG: JSON can be indexed in Postgres You could also support a hybrid JSON/Relational model giving the best of worlds.
Read:
Create views into the JSON schema that model your READ access patterns and expose them as IMMUTABLE Relational entities. (Clearly they should be as light weight as possible)
Modify:
You can split the JSON blobs into their own skinny tables. This should keep your current semantics and facilitate faster targeted updating.
Big blobby resources such as video/audio should be managed as resources and not junk up your DB
Warning:
Abstracting the model into multiple tables may cause its own issues depending on how you ORM map your entities.
Outside the Box Thinking:
-Extract and transform the data for optimized reading. -Move to MongoDB or a Key Value store
Conclusion:
What are the update patterns? -is only 1 field being updated -inter-dependencies of the data being updated How are "update anomalies" minimized
You will need to create a migration strategy to a more optimal solution and would do well to start abstracting with views. As the data model is improved this will be a continuous process and the data model can be "optimized" without disturbing the core infrastructure requiring rewrites.
As someone said, indexes are the best to do lookups. Remember your DB engine do lookups internally even if you are not aware (joins, for example), so add indexes to join fields.
Another thing that worked me for me (and I dont know if it's your case), was to add trigram text indexes, which make it faster to do a full text search. Remember, anyway, that adding a index makes search faster, but insert slower, so be careful if you are inserting a lot of data.
Rearchitecting the schema might be worth doing. From the technical side, PG is pretty nice about doing transactional schema changes. I'd be more worried about the data though. Are you sure that every single row's Json columns have the keys and value types that you expect? Usually in this type of database, some records will be weird in unexpected ways. You'll need to find and account for them all before you can migrate over to a stricter schema. And do any of them have extra unexpected data?
- Change the field type from JSON to JSONB (better storage and the rest) https://www.postgresql.org/docs/13/datatype-json.html
-Learn the json in-build functions and see if one of them can replace one you made ad-hoc
- Seriously, replace json with normal tables for the most common stuff. That alone will speed up things massively. Maybe keep the old json around in case, but remove when it become old(?)
- Use views. Views allow to abstract over your databaee and allow to change internals
- If a big things is searching and that searching is nkind of complex/flexible, add FTS with proper indexing to your json then use it as first filter layer:
This speeeeeedupppp beautifully! (I get sub-second queries!)- If your query do heavy calculations and your query planner show it, consider move them into a trigger and write the solved result into a table, and query that table instead. I need to loans calculations that requiere sub-second answers and this is how I solve it.
And for your query planner investigation, this handy tool is great:
https://tatiyants.com/pev/#/plans
An hour with the query planner can save you days or weeks or wasted work!
I currently have a UI with the comments down the side of the screen which looks like this:
https://www.volt.school/videos/c980297a-417b-416f-947b-58a70...
This is good because you can easily: - See all the comments - Navigate between them - See replies etc.
However it has a huge problem with you trying to balance watching the video with reading the comments.
I also have an alternative UI I've been working on which only shows one comment at a time:
https://www.volt.school/videos-v2/c980297a-417b-416f-947b-58...
However the downsides of this is that you can't see all the comments at once. I'm not a UI/UX designer AT ALL so I'd really appreciate some pointers around how to think about making this better! The original post mentions "close to solving", I think I am pretty close but it's still not quite right and while I'm not out of ideas yet, I'd appreciate feedback if solving this is obvious to someone else.
How about breaking the play into chapters/zones/rooms/segments (whatever makes sense for the game) then showing all the comments for that segment. Once the segment ends, there would be a replay button if they missed anything on the first play while reading comments, and a next segment button to carry on.
Interesting time spans could be marked for slow motion, boring bits played at double time.
There would be high level navigation between segments with thumbnails and comment counts. Buttons to skip between “pivotal moments”, maybe with voting to highlight them.
Something like Soundcloud comments but for video.
Asian video platforms used to do that. Here's an example: https://www.youtube.com/watch?v=hOMMQmYwd4I
It's totally crazy but can be made much more coherent. Would be useful to have comments at a specific time and place on the screen just for very accurate pointers/comments.
I don't know how normal my use is though or if that's at all helpful.
Any help would be greatly appreciated.
Hawking radiation ?
Laser tunnel ?
Magnetic canon ?
Centrifugal launcher ?
Vacuum diffusion ?
Electrical beam lensing ?
That actually does make for a fairly efficient (better than fusion in energy per unit fuel mass) reactor design in principle, but you need a sub-solar-mass black hole (in the 1 billion - 100 billion ton range), and there's no known practical way to produce one.
... Not being flippant; I find that these kind of prompts can help in thinking from a new approach.
A huge bonus there would be when the order difference can be represented in a graph, so that tesselation or other approaches like a hypercube representation can be used for quick estimations. (that's what I'm aiming for right now)
If successful, the next step would be to integrate it into my web browser so that I can try out whether the equilibrium works as expected on some niche topics or forums.
[1] https://arxiv.org/abs/1603.04641
1. Ritualize
I notice a pattern with my kids: going to bed is a ritual, and any deviation is reason enough for them to leave the bed.
2. Slow down in advance
Going to bed after playtime is impossible. So cut screens, playtime, play/listen some quiet music, read books or newspaper (again, no tablet/reader), at least 1h before bed time.
3. Recap the day
Remind your kids of their day, activities and make them aware of the fatigue. Works better when in bed, with mine.
4. Stay with them if they're afraid
Learn why they're afraid, teach them why there's no reason to be afraid. I've had to hang a sock to the door every night for months to scare tigers away :D It just works ^^
Every parent knows the pain and every kid has their own back story and the relationship with the parent(s) is key to finding a way into bed.
Eventually, they will sleep.
In our case, we settled that unless exceptional situations, our kids had to fall asleep in their own bed, because we wanted/needed our intimacy. To get that, I had to stay in my kids' room for as long as 2hrs for months, but didn't let go. Today, going to bed is thankfully not a situation anymore.
I'm not sure this is in any way helpful, but here's my shared experience and learnings. YMMV.
So for us, when I first started to do this. Each night they get a 'treat' but to get that treat they first need to be ready for bed - eg bedroom ready to sleep in, correctly dressed/ washed etc..
then after the treat they must choose a calming activity - ideally in their bedroom eg reading (nothing that gets their heart rate up) for 30-60mins then they must bush their teeth
It's this point we say time for bed, but we allow them to carry on reading for another 30-60min then it's lights off
if they don't do the activities/ actions after the treat, then we warn them that they'll not get one tomorrow etc. (and really do what you say)
also you may need to flexible on the activities until they get into the swing of it
https://xkcd.com/320/
(The only thing worse than trying to get a child who isn't tired to go to sleep is a child who is too tired to go to sleep.)
Apparently I also still have a sense of humor. Or maybe I don't, because perhaps pretending one doesn't get the joke of doubling down on xkcd silliness is perhaps a joke in itself, which I didn't get.
Also, bedtime trouble usually means not enough outside time and physical activity during the day. Or that the kids want more of the parent's time.
I realize that some parents reach for it every night and this is not something I'm suggesting.
https://www.amazon.com/Bringing-Up-B%C3%A9b%C3%A9-Discovers-...
1. make sure they don't fall asleep with something they cannot keep all night (i.e. while you are singing to them, rocking them, sitting next to them or when they are drinking a bottle of milk etc)
2. make sure they understand that even if you leave the room it is just temporarily. Small kids are - for good reasons - very afraid of being forgotten or left alone.
2.1 Using a timer to remember to visit the room regularly and often as they learn to sleep alone can help a lot
2.2. Increase the interval each day. I increased it by two minutes each day.
2.3 If the kids are happy in their bed, continue to visit their room at the scheduled time: you don't want them to think that you forget them if they don't cry.
Using this method I've got my last few kids to enjoy going to bed and sleep better in less than a week fo each of them.
I am looking for my first client: Ideally someone in charge of a Museum/gallery or other grandiose indoor space.
I wish AirBnBs and hotel rooms would offer this type of preview of their premises.
Edit: There's a bug where if you start dragging with the mouse and let go the mouse button outside the 3D view, it acts like the button is still held down (a bit like mouse capture) which was easier but quite confusing.
It seems unusable with a touch screen on desktop.
The fact that you can fly is useful but non-obvious. I ended up down at floor level and wondering how to see the pictures on the walls.
The demo is in a /fr/ path, but is in English (Chrome offers to translate it to English, because it somehow thinks the English words are French), but then some parts of the interface like "Share your place" are in French.
Additionally, I think it would be best if by default you were stuck to standing head height, then you can either provide buttons to actually move up or down, or lean more into the game aspect and allow the user to jump. Right now it feels like you are floating around with a little drone or something.
In the vein of controls, please please please support WASD too. I understand if you instruct with arrow keys, since for non-gamer users it might be more obvious, but support WASD (or equivalent of what WASD is in QWERTY keyboards) anyways, for 2 reasons: it is much more ergonomic for people who use a mouse on their right hand (I myself am left-handed but use the right hand for mouse anyways), and is more ergonomic for some laptop users, since many laptops have half-size vertical arrow keys which are uncomfortable to press all but momentarily.
As for the head, yes you are right : I should add 'something' that tilts a bit the head up/down when needed.
1. Your demo level suffers from Z-fighting in a few places on the floors and walls. https://en.wikipedia.org/wiki/Z-fighting
2. When viewed full-screen on a 4k monitor, textures are too low resolution. A handwritten note on the wall is unreadable.
3. Lighting is too simple. Because that’s not an FPS shooter you probably don’t need dynamic lightning nor day/night cycle, but it’s still hard. Ideally you need these multiple PBR textures everywhere, and correspondingly complicated pixel shaders.
2: yes but it is a tradeoff between texture quality and minimazing loading time.
3: Yea, ideally. But KISS is my priority : We have an editor that aims to be simple enough for all, thus no BPR & no shader.
3 — I see. Still, you could pre-compute local illumination automatically in the editor, and bake it somewhere. Maybe into vertex attributes, maybe into another lower-resolution R8_UNORM set of textures.
3- --> I will see with client feedback. I do not want at this point to over-engeneer free-visit. First I must find my market.
I'm curious on how other people solved it ( by cookies, subdomain, ... ) and if you used a JwtToken for it.
But I haven't decided yet on the actually flow. Where I'd identify the current tenant or impersonate him.
+ The influence of impersonation on that flow.
Related: I'd love to have an Android app with a shortcut that allows me to quickly translate Google Maps links into coordinates, OSM links or other map links. There is a browser extension that does this on desktop, so if anyone is looking for a low hanging fruit idea for an Android app, this might be a fun idea (if I don't get around to it first).
Everything is seamless for me, though admittedly I'm not a super heavy calendar user.
I plan to do a write up on my whole Google-free setup, but I haven't done it yet, unfortunately.
I am starting to consider alternative tools such us wireguard to reduce load, but I am concerned of adding too much complexity. Tinc's mesh network makes setup and maintenance easy. The wireguard ecosystem seems to be growing very quickly, and it's possible to find tools that aim to simplify its deployment, but it's hard to see which of these tools are here to stay, and which will be replaced in a few months.
What is the best practice, in 2021, to ensure all communication between cloud VMs (even in a private network) is encrypted?
[0]https://www.tinc-vpn.org/
DIY: envoyproxy.io / HashiCorp Consul for app-space private networking over public interfaces.
LowCode: Mesh P2P VPN network among your clusters with FOSS/SaaS like WireTrustee / tailscale.io / Slack Nebula.
Outside of the WireGuard ecosystem there's ZeroTier [2] which has been around for a while and they're working on a new version; and Nebula [3] from Slack, which is likely to be maintained as long as Slack uses it.
There might be others, but with tinc these four are the ones I've seen referred to most often.
[1] https://tailscale.com
[2] https://www.zerotier.com
[3] https://github.com/slackhq/nebula
[1] https://github.com/tonarino/innernet
Have you noticed whether it is worse for lots of small requests vs large data transfers?
I use a very similar setup, but haven't seen tinc CPU usage matter yet, though for very low traffic.
I’m considering moving the binary data into S3 and then doing the sync layer on the server (which means the front end requests the data from the backend and is given it back as a JSON object with base64 values).
Doing this manually via code isn’t impossible, just API intensive, so I’m wondering if this is a solved issue for anyone.
The why: The JSON blobs are recordings of words and sentences that can be copied between articles.
Where/why is your current system failing/inadequate/cumbersome?
Why do you want to move the data to s3?
Why are delete cascades important?
We are on JIRA now, and it’s … JIRA. We tried basically any other tool, including Excel (yes, that is somewhat possible).
My problem generally is that tools are slow, planning is cumbersome, visibility is limited and reporting for clients is often even more limited.
Heck, I’d even write my own tool if I knew it would help others, but I am concerned it’s too close to what we already have for anyone to actually migrate.
You could help me by sharing your thoughts!
You could also modern try agile tools, for example Linear. JIRA is good for 100+ teams and complex architectures.
Not affiliated, but I've had a positive experience with it in a small team. I would describe it as an IDE for issues.
https://clickup.com https://clickup.com/on-demand-demo
ClickUp for Agile Workflows https://www.youtube.com/watch?v=H9hZRwivnL8
1. I want to create way to generate electrical power without pollution. Basically, a closed cycle process that releases no pollutants, or electronic waste.
2. I want to do everything I can to eliminate gender bias in the world.
We have a small hydro-electrical plant una River near my house and really it's no big deal, it fits very nicely in the surrounding environment and it produces clean energy.
It's also educational because since the river is near the city small children classes can visit it and learn about it.
http://ludens.cl/paradise/turbine/turbine.html
Geothermal can be a solution for generating electricity directly, but if you'd like to minimize electronic waste perhaps it would be easier to use it to replace alternative energy sources for HVAC purposes.
Biofuels (eg: plant bamboo, grow it, then burn it) can also technically be closed cycle energy sources.
Solar water heaters can also reduce electrical or fossil-fuel-based energy consumed for generating hot water.
What progress have you made in your work on either front? What sort of work do you do to solve these problems?
I am also spending much time lately in the SF kink community to build a fundamental understanding of the biases people have experienced in life with respect to their gender identity, and am strongly considering HRT so I can live life on the other side and experience the prejudice first hand.
You can help by trying the API and giving honest feedback.
https://bruzu.com
1. Image generation automation: Like into automation like posting tweets as image to instagram.
2. Image generation scaling: Create multiple images with just variable text, like greeting messages or product images or open graph images.
Some feedback on the Designer:
The size setting dropdown is quite strange. Choices of unfamiliar destinations don't seem to make sense, and some of the things that do seem familiar come out an unexpected size and shape (e.g. "infographic"). The pixel sizes are clearer, but better would be handles on the canvas that can be dragged. There's also a typo: "Choose form a list of sizes".
Circles don't get resized? I can drag handles to make the apparent bounding box bigger, but the circle doesn't change: https://imgur.com/a/KhYluKj. Other shapes seem okay. Chrome 89 on Linux.
The tutorial walkthrough pops up every time I go to the designer, even though I've been right through it.
Fixed most of it.
Your service could provide pre-made templates and an editor, and expose textfields, images, fonts, etc options via URL parameters. Then your service just has to render the SVG and return it as an image with the requested dimensions/format.
Example:
`https://img.bruzu.com?s=<TEMPLATE ID>&title=Hello&font=arial&width=800&height=480&fmt=png`
You could also pass the raw SVG source as a parameter as well, maybe with base 64 encoding or something like that.
Of course google can't do it. But this is a ripe for someone to step in.
Basically, every leafy green (and herbs, and even mushrooms), can grow in a range of climatic condition (phenotype, roughly) ie temperature, humidity, water, CO2 level, pH, light (spectrum, duration and intensity) etc. As you might have seen around the world there is a rise in indoor vertical farms, but the truth is that 50% of those are not even profitable. My startup wants to discover the optimal parameters for each plant grown in our indoor vertical farm and eventually I would let our AI system control everything (something like alphaGo, but for growing plant X (lettuce, kale, chard, ). Think of it as reinforcement learning with live plants! I am betting on the fact that our startup will discover the 'plant recipes' and figure out the optimal parameters for the produce that we would grow. Then, the goal is that cities can grow food cheaper in more secure and sustainable way than our 'outsourced' approach in country side or far away lands.
So now I have secured some funding to be able to start working on optimizations, but I realized that *hardware* startups are such a different kind of beast (I am a good software product dev though, I think). Honestly, if anyone with experience in hardware related startups (or experience in the kind of venture I am in) would just want to meet me and advise me, I would take it any day. Being the star of the show, it's hard for me to handle market segmentation, tech dev, team, next round of funding, European tech landscape, etc. I am foreseeing so many ways that our decisions can kill my startup, all I need is advise from someone qualified/experienced enough. My email: david[at]hexafarms.com
At the very least what's a link to your startup's website?
If you want users to have it from your profile, put it in the “about” field.
Sounds similar to what I read a long time ago about a big tomato farm in the Netherlands... Have you tried talking to actual farmers of that produce? Universities? Agricultural faculties do a lot of research in that direction.
Expensive, quickly perishable produce might be able to compete, otherwise I guess free water and energy from above in the "remote" classical farming will be hard to beat.
And then my naive guess would be that to generate enough data for a "ml" approach not only by name might be somewhat expensive.
This sounds so negative, but this is not my intention... I wish you all the best and hopefully will stumble upon a success story in the future :-)
I heard about a greenhouse company that has programmed their climate control to match “best growing conditions historical weather”. So, they ask local experts what year / location had the best X and then they use that region’s historical weather and replay it in their greenhouse. I thought that was brilliant!
(Just realized this was Kimbal Musk that mentioned this)
This had replaced shortening hormones in modern gardening (or at least at that greenhouse, but my understanding they were just doing the same thing as everyone else).
I guess there is a lot more to learn for those who have scale enough to experiment and patience to follow through.
I agree that micronutrient content has decreased in the past century. Some might be because of scale, some might be that yield gains are mostly driven by macronutrients and water, not micronutrients, it could be selecting varieties that taste better, or it could be depleting the soil.
That said, the US has an obesity epidemic, so there's no shortage of macronutrients. Macronutrient shortages also seem rare. Scurvy and rickets aren't exactly problems.
The process might span different medium (write email, do something in the app, check twitter, etc) and different activities multiple days. How to make sure they know what they should do next? Checklist? Emails? Slack? Wizard?
I have a plugin that exports some WooCommerce orders into XLS. I would like to add a progress bar via AJAX, because the export may take very long for thousands of orders. But I am not really sure how to use AJAX in context of Wordpress specifically.
I would love to see a minimal functional example, a simple plugin that does something similar. So far, all the plugins I saw were pretty convoluted and I lost my track around the code.
(On a related note: a library of elementary examples for Wordpress plugin development would be nice. Like "This is how you create a menu entry.")
I guess you already have an URL on your Wordpress setup that triggers this export. Let's call it {url}/export.
Wordpress already has jQuery by default included. So'll you need to call that URL using jQuery $.post and then, accordingly to the response, update your progress bar.
There is nothing specifically about Wordpress on this, besides the fact that you need to setup your own URL on Wordpress to do this, and then include your own JS after jQuery. That's all.
If you find this too-complicated, a quick-hack is to create a page on WP Admin called Export Tool, and then on your theme create page-export-tool.php. That .php will be called when visited that Export Tool page.
By web search I found this to tutorial to put sentences in an embedding space: https://github.com/BramVanroy/bert-for-inference/blob/master...
I did not read this and am not endorsing it, but it looks like it’s doing roughly what I’m suggesting.
Your idea, if implemented well, may end up being a net positive for society, but I can't help imagining a future where every child, from the moment they are born, has a biometric ID connecting them to a consortium of companies which provide their education, health care, housing, energy, internet connectivity, transport, media access, and so on.
It would be like living in a company town, being paid in company scrip, except you wouldn't notice the restrictions (as long as you kept earning). If you ever increased your income, your consortium might let you choose whether you want to upgrade your housing or your health care plan, but if you lost your job, they'd force you to take one of their choosing and downgrade your plans if it had a lower salary.
In this dystopia, all consumables from food to toilet paper would presumably be sold by Amazon, and other items like furniture and electronics would be provided as a service so that you rent them from your consortium. The only question is why people wouldn't try to undo this system through the political process, but then we might ask that about the current system.
Thank you!
> but I can't help imagining a future where every child, from the moment they are born, has a biometric ID connecting them to a consortium of companies which provide their education, health care, housing, energy, internet connectivity, transport, media access, and so on.
My idea will not have that side-effect because not only is the project non-profit and open source, but it is also decentralised. And if we keep thinkng of a dystopian future then we won't be able to do anything positive unless we become some sort of social revolutionaries. I don't have those skills. But I can think of small ideas to make a positive impact that benefits everyone though. The idea I listed in my original post above is very simple, it helps teachers, lecturers, or anyone who contibutes to a persons monetary decent life via education get appropriately paid for their efforts, and everyone(i.e. businesses) who benefits from an educated person should contribute towards that.
It is a simple idea but notoriously diffcult to deploy because there is a possibility of this getting caught up in a political slugfest.
* I'd like to donate my kidneys, liver, and lungs as a living donor. However I've heard that there may be a psychological screening component for being a living donor which may detect that I'm planing on killing myself and trigger intervention. Is there a way around it? It's not critical for success but it would be nice to give back to someone else a chance at life, but I'm not sure if it's doable.
* I want to minimize the possibility of being discovered and identified after death. I've selected a heavily wooded site that's time consuming to access on foot and selected timing to be early autumn some years from now once I've acclimated my acquaintances to my absence. The concern however comes from the fact that I have a security clearance and my fingerprints are in the system. I'm trying to figure out if it's a real concern that my body is discovered with fingers intact, and what needs to be done to mitigate it. Or if seclusion, exposure, and scavengers are enough to cover that possibility and it's not a real concern.
Reasoning: helping with the code behind a paper on explanatory AI systems.
Related code on github if curious https://github.com/pollomarzo/map-generation/tree/main/graph
EDIT: thanks for suggestions will give them a spin throughout tomorrow :)
Iframes with postmessage() where needed (like dynamic window size changes) isn't pretty, but it's easy to do.