Just for you, I went through my logs for my current irc server and checked how often irc messages occured on the same millisecond. I found on the order of a few thousand such events out of millions of lines total.
In addition, that's just based on my perception; IRC is NOT an ordered protocol, so if multiple people ran this same logger, they'd end up with different stream and different duplicate keys (same thing as events lost I might add).
My log, also, conveniently contains joins, parts, notices, ctcps, and all that other jazz because I used a mature logging library.
file.write is atomic (guaranteed to "work") for PIPE_BUF (posix) octects ~ 16ko at most.
JSON is guaranteed to be unparsable if the file is truncated of the last chars (it is not very resilient).
Hence the write may corrupt your WHOLE log if a non recuperable failure happens or the code is interrupted.
The code DOES not use fixed size allocation ... thus is can crash randomly because of SEGFAULT in the middle of the writing.
The history will take cumulative size in memory.
This coding attitude highers the probability of this failure to happen BY DESIGN.
Is it that complex to FIRST write the file, and THEN atomically rename the new file to the old file at worst (resulting only in losing the current session, but not the whole history).
If your log are that precious, why would you not take extra care about protecting them?
This code makes me want to puke.
On the other hand, it is representative of the reason why I am disenchanted by modern coding standards.
Not Julie. It is a pun in french. Julie is like calling me Hanni when my real name is more like Hanniballs. And I still have my balls. So keep it julie1.
But I guess you may lack of context to get it right, as for the rest of your comment. ;)
Logs should only be built by chunking at the end of the file atomic write of data in at most PIPE_BUF with an "atom" based on this quantity of data. You are free to do whatever serialization inside this constraint as long as a missing chunk cannot corrupt the whole file.
(That is the reason I switch off from systemd that stopped respecting this rule.)
For your logger to work in a "safe fashion" it has to be designed like an asynchronous HW driver.
A part consuming the logs from IRC, that pushes chunks in circular buffer ring of N fixed size data for which buffer overflow HAVE to be checked (per chunk size and for N) (the bottom half). And a driver that is signaled "asynchronuously" to write the chunks into a file when ready. The critical part (writer) should almost be a one liner to avoid interruption and be written in an overly defensive paranoid fashion.
You should have a cursor for the writer and the reader on the circular buffer ring and should always check that the consumer does not overrun the writer. Hence the writers' code should be designed to be asymmetrically faster than the reader.
It requires IPC or messaging to coordinate both reader and writer. Most people do it in multithreading, I do it with multiprocessing because it is simpler. You should understand that you have thus a Single Point Of Failure (the file) and that because of all this your software will have a domain in which it will fail (if you want to multiply the reader to distribute the load for the consumer to 'scale up').
The file has to be opened with exclusive access. The code must have a "catch all I can" to be able to unlock the file if ever something bad happens and at startup the code should check it can open the file. (atexit callbacks and every "recoverable" interruptions sent by the OS).
file.seek is your friend. You should not use any dynamic allocation to make it more robust (yes preallocating fixed memory size with "char circular_ring[N * MAX_SIZE]")
Where N and MAX_SIZE are fixed.
Well. It requires a complete rewrite for this code to not create a risk for your users. Your code is like full of hidden landmine by lack of design.
And you know what?
It used to be standard knowledge for introduction to CS for scientific. You know why? Because no users of computers like to lose their precious data made during a very long measurement spilling huge amount of data and costing a lot of resources. (like gold, helium, molybden, electricity, time, wages of qualified operators....)
It seems like coders do not care of their users, like a builder thinking it is reasonnable to build 50 store buildings on quicksands because people only judges builder by the look of their creation.
Part 1/2: Resurrecting SFJulie1's original comment here because it seems like it was unfairly killed:
Ohla. You have to rewrite it entirely.
Logs should only be built by chunking at the end of the file atomic write of data in at most PIPE_BUF with an "atom" based on this quantity of data. You are free to do whatever serialization inside this constraint as long as a missing chunk cannot corrupt the whole file.
(That is the reason I switch off from systemd that stopped respecting this rule.)
For your logger to work in a "safe fashion" it has to be designed like an asynchronous HW driver.
A part consuming the logs from IRC, that pushes chunks in circular buffer ring of N fixed size data for which buffer overflow HAVE to be checked (per chunk size and for N) (the bottom half). And a driver that is signaled "asynchronuously" to write the chunks into a file when ready. The critical part (writer) should almost be a one liner to avoid interruption and be written in an overly defensive paranoid fashion.
You should have a cursor for the writer and the reader on the circular buffer ring and should always check that the consumer does not overrun the writer. Hence the writers' code should be designed to be asymmetrically faster than the reader.
It requires IPC or messaging to coordinate both reader and writer. Most people do it in multithreading, I do it with multiprocessing because it is simpler. You should understand that you have thus a Single Point Of Failure (the file) and that because of all this your software will have a domain in which it will fail (if you want to multiply the reader to distribute the load for the consumer to 'scale up').
The file has to be opened with exclusive access. The code must have a "catch all I can" to be able to unlock the file if ever something bad happens and at startup the code should check it can open the file. (atexit callbacks and every "recoverable" interruptions sent by the OS).
> How can a developer be that stupid? [...] This code makes me want to puke.
This comment breaks the HN guidelines so badly that we could put it on a poster of how never to behave here. It doesn't matter how right you are if you express it this abusively.
Since you've done this more than once before, I've banned your account. If you don't want it to be banned, you're welcome to email hn@ycombinator.com and give us reason to believe that your comments will be civil in the future.
var array = [];
client.addListener('message', function (from, to, text) {
var time = Math.floor(new Date() / 1000);
var obj = { nick: from, message: text };
var item = '\"' + time + '\"\:' + JSON.stringify(obj);
array.push(item);
fs.writeFile(file, '{' + array + '}\n', function() {
// console.log("updated");
});
});
This will buffer all the messages in memory (in the variable "array") and rewrite the file on every message. Besides the lack of atomicity mentioned in another comment, that's also really inefficient and will eventually run out of memory. Plus 1000 incoming messages will lead to 1000 sequential rewrites of the same file!
A useful file format to deal with this kind of problem is JSONL, which is just a list of newline separated json blobs. You can then simply append things to the file rather than having to write the whole thing each time.
Please don't "Show HN" something that is neither complicated nor interesting.
The other "Show HN" on the front page right now, for comparison, is Claudia.js. That's not something anyone on HN could trivially write in 5 minutes. It's about 5k lines of javascript. It's already somewhat mature.
For comparison, the project you posted is under 50loc, any of us could trivially write (in fact, piping http://tools.suckless.org/sic/ would be sufficient... or using znc with its "log" module, or using weechat / irssi's builtin logging functions), and doesn't really accomplish something super interesting to HN imo.
I expect everyone who uses IRC in a significant capacity on HN has already solved the most basic level of their logging problem. I solved mine by just running a ZNC bouncer with the log module.
The harder level of the logging problem, indexing it and presenting it in a nice UI and solving availability by merging logs from multiple leafs in a netsplit event, I'm not so you can consider commonly solved, but hey, your solution has nothing to say there either.
Others have already mentioned that there are some issues with your code, so I won't touch on them, but I will emphasize that you should probably work on improving code quality and make something both more significant and more interesting prior to doing a Show HN.
In addition, this actually is something people on HN actively couldn't trivially use because it has no configuration (hard coded connection strings) and no docs.
Let's say that you're an irc network operator trying to make the channel's entire history searchable (e.g. botbot.me type thing).
For absolute correctness, you'd want to record per-leaf and merge.
However, the last time I needed log-merging functionality was much more boring; some of my log-files were corrupted, and someone else had logfiles that weren't as extensive, thus I needed to munge the two together. The timings were subtly different (because that's how irc works), so I wrote custom code to munge them together, preferring his for the range of corruption.
So yes, I've needed that functionality, though it wasn't actually done due to a netsplit. I can foresee it being useful in the case of a netsplit if your logging is at the ircd level or you run one bot per leaf.
I wonder whether any large network keeps logs of channels, let alone one per server. And if they do, they probably don't have much use for a merged log except for taking up less space.
My own IRC log tool [GH: tilpner/ilc] can be used to merge logs, but I rarely use that functionality.
For two log files "a" and "b", in weechats default log format:
ilc sort -f weechat -i <(cat a b) | ilc dedup -f weechat
I've never tested this with logs over 200MB, but sort will read the combined log into memory, which is definitely not optimal.
Don't use this prettified JSON format. It's awful to parse. Put each message on it's own line. I had trouble parsing someone's irc logs in JSON because it was too big to load into memory. It was much easier to just load one line at a time from the file.
file.seek is your friend. You should not use any dynamic allocation to make it more robust (yes preallocating fixed memory size with "char circular_ring[N * MAX_SIZE]")
Where N and MAX_SIZE are fixed.
Well. It requires a complete rewrite for this code to not create a risk for your users. Your code is like full of hidden landmine by lack of design.
And you know what?
It used to be standard knowledge for introduction to CS for scientific. You know why? Because no users of computers like to lose their precious data made during a very long measurement spilling huge amount of data and costing a lot of resources. (like gold, helium, molybden, electricity, time, wages of qualified operators....)
It seems like coders do not care of their users, like a builder thinking it is reasonable to build 50 store buildings on quicksands because people only judges builder by the look of their creation.
Click on the comment's timestamp to go to its page, then click 'vouch'. But note that you need more than 30 karma before the 'vouch' and 'flag' links appear.
26 comments
[ 4.7 ms ] story [ 60.3 ms ] threadI'd much prefer an array with objects as element, each object storing its own timestamp.
In addition, that's just based on my perception; IRC is NOT an ordered protocol, so if multiple people ran this same logger, they'd end up with different stream and different duplicate keys (same thing as events lost I might add).
My log, also, conveniently contains joins, parts, notices, ctcps, and all that other jazz because I used a mature logging library.
https://github.com/egladman/herodotus/blob/master/server.js#...
How can a developer be that stupid?
file.write is atomic (guaranteed to "work") for PIPE_BUF (posix) octects ~ 16ko at most.
JSON is guaranteed to be unparsable if the file is truncated of the last chars (it is not very resilient).
Hence the write may corrupt your WHOLE log if a non recuperable failure happens or the code is interrupted.
The code DOES not use fixed size allocation ... thus is can crash randomly because of SEGFAULT in the middle of the writing. The history will take cumulative size in memory.
This coding attitude highers the probability of this failure to happen BY DESIGN.
Is it that complex to FIRST write the file, and THEN atomically rename the new file to the old file at worst (resulting only in losing the current session, but not the whole history).
If your log are that precious, why would you not take extra care about protecting them?
This code makes me want to puke.
On the other hand, it is representative of the reason why I am disenchanted by modern coding standards.
Sometimes some practice reflect the total lack of potential professionalism.
Before making money out of food, you care about not poisoning your customers.
And any cook refusing to respect hygiene rules should be barred from kitchen by any sane managers, because there is nothing you can do with them.
Same goes with some coders.
Do you live in a state of perpetual nausea? Because there is so much code out there like this...I fear it could overwhelm your life to worry about it.
I think the best we can do is just fix the stuff we want to use and ignore the rest..
But I guess you may lack of context to get it right, as for the rest of your comment. ;)
Logs should only be built by chunking at the end of the file atomic write of data in at most PIPE_BUF with an "atom" based on this quantity of data. You are free to do whatever serialization inside this constraint as long as a missing chunk cannot corrupt the whole file.
(That is the reason I switch off from systemd that stopped respecting this rule.)
For your logger to work in a "safe fashion" it has to be designed like an asynchronous HW driver.
A part consuming the logs from IRC, that pushes chunks in circular buffer ring of N fixed size data for which buffer overflow HAVE to be checked (per chunk size and for N) (the bottom half). And a driver that is signaled "asynchronuously" to write the chunks into a file when ready. The critical part (writer) should almost be a one liner to avoid interruption and be written in an overly defensive paranoid fashion.
You should have a cursor for the writer and the reader on the circular buffer ring and should always check that the consumer does not overrun the writer. Hence the writers' code should be designed to be asymmetrically faster than the reader.
It requires IPC or messaging to coordinate both reader and writer. Most people do it in multithreading, I do it with multiprocessing because it is simpler. You should understand that you have thus a Single Point Of Failure (the file) and that because of all this your software will have a domain in which it will fail (if you want to multiply the reader to distribute the load for the consumer to 'scale up').
The file has to be opened with exclusive access. The code must have a "catch all I can" to be able to unlock the file if ever something bad happens and at startup the code should check it can open the file. (atexit callbacks and every "recoverable" interruptions sent by the OS).
file.seek is your friend. You should not use any dynamic allocation to make it more robust (yes preallocating fixed memory size with "char circular_ring[N * MAX_SIZE]")
Where N and MAX_SIZE are fixed.
Well. It requires a complete rewrite for this code to not create a risk for your users. Your code is like full of hidden landmine by lack of design.
And you know what?
It used to be standard knowledge for introduction to CS for scientific. You know why? Because no users of computers like to lose their precious data made during a very long measurement spilling huge amount of data and costing a lot of resources. (like gold, helium, molybden, electricity, time, wages of qualified operators....)
It seems like coders do not care of their users, like a builder thinking it is reasonnable to build 50 store buildings on quicksands because people only judges builder by the look of their creation.
Ohla. You have to rewrite it entirely. Logs should only be built by chunking at the end of the file atomic write of data in at most PIPE_BUF with an "atom" based on this quantity of data. You are free to do whatever serialization inside this constraint as long as a missing chunk cannot corrupt the whole file.
(That is the reason I switch off from systemd that stopped respecting this rule.)
For your logger to work in a "safe fashion" it has to be designed like an asynchronous HW driver.
A part consuming the logs from IRC, that pushes chunks in circular buffer ring of N fixed size data for which buffer overflow HAVE to be checked (per chunk size and for N) (the bottom half). And a driver that is signaled "asynchronuously" to write the chunks into a file when ready. The critical part (writer) should almost be a one liner to avoid interruption and be written in an overly defensive paranoid fashion.
You should have a cursor for the writer and the reader on the circular buffer ring and should always check that the consumer does not overrun the writer. Hence the writers' code should be designed to be asymmetrically faster than the reader.
It requires IPC or messaging to coordinate both reader and writer. Most people do it in multithreading, I do it with multiprocessing because it is simpler. You should understand that you have thus a Single Point Of Failure (the file) and that because of all this your software will have a domain in which it will fail (if you want to multiply the reader to distribute the load for the consumer to 'scale up').
The file has to be opened with exclusive access. The code must have a "catch all I can" to be able to unlock the file if ever something bad happens and at startup the code should check it can open the file. (atexit callbacks and every "recoverable" interruptions sent by the OS).
This comment breaks the HN guidelines so badly that we could put it on a poster of how never to behave here. It doesn't matter how right you are if you express it this abusively.
Since you've done this more than once before, I've banned your account. If you don't want it to be banned, you're welcome to email hn@ycombinator.com and give us reason to believe that your comments will be civil in the future.
All: In addition to https://news.ycombinator.com/newsguidelines.html, please know that HN has extra rules at https://news.ycombinator.com/showhn.html to guide the discussion of new work. The parent comment breaks every one of them.
The other "Show HN" on the front page right now, for comparison, is Claudia.js. That's not something anyone on HN could trivially write in 5 minutes. It's about 5k lines of javascript. It's already somewhat mature.
For comparison, the project you posted is under 50loc, any of us could trivially write (in fact, piping http://tools.suckless.org/sic/ would be sufficient... or using znc with its "log" module, or using weechat / irssi's builtin logging functions), and doesn't really accomplish something super interesting to HN imo.
I expect everyone who uses IRC in a significant capacity on HN has already solved the most basic level of their logging problem. I solved mine by just running a ZNC bouncer with the log module. The harder level of the logging problem, indexing it and presenting it in a nice UI and solving availability by merging logs from multiple leafs in a netsplit event, I'm not so you can consider commonly solved, but hey, your solution has nothing to say there either.
Others have already mentioned that there are some issues with your code, so I won't touch on them, but I will emphasize that you should probably work on improving code quality and make something both more significant and more interesting prior to doing a Show HN.
In addition, this actually is something people on HN actively couldn't trivially use because it has no configuration (hard coded connection strings) and no docs.
That'd require one log/connection per server of the network, which isn't something ZNC or weechat will do by default.
Have you ever needed that functionality?
For absolute correctness, you'd want to record per-leaf and merge.
However, the last time I needed log-merging functionality was much more boring; some of my log-files were corrupted, and someone else had logfiles that weren't as extensive, thus I needed to munge the two together. The timings were subtly different (because that's how irc works), so I wrote custom code to munge them together, preferring his for the range of corruption.
So yes, I've needed that functionality, though it wasn't actually done due to a netsplit. I can foresee it being useful in the case of a netsplit if your logging is at the ircd level or you run one bot per leaf.
My own IRC log tool [GH: tilpner/ilc] can be used to merge logs, but I rarely use that functionality.
For two log files "a" and "b", in weechats default log format:
I've never tested this with logs over 200MB, but sort will read the combined log into memory, which is definitely not optimal.The bar for Show HN doesn't have to be that high. It's ok for people to post projects that they're working on while learning something, for example.
https://news.ycombinator.com/showhn.html
file.seek is your friend. You should not use any dynamic allocation to make it more robust (yes preallocating fixed memory size with "char circular_ring[N * MAX_SIZE]") Where N and MAX_SIZE are fixed.
Well. It requires a complete rewrite for this code to not create a risk for your users. Your code is like full of hidden landmine by lack of design.
And you know what?
It used to be standard knowledge for introduction to CS for scientific. You know why? Because no users of computers like to lose their precious data made during a very long measurement spilling huge amount of data and costing a lot of resources. (like gold, helium, molybden, electricity, time, wages of qualified operators....)
It seems like coders do not care of their users, like a builder thinking it is reasonable to build 50 store buildings on quicksands because people only judges builder by the look of their creation.
We banned SFjulie1: https://news.ycombinator.com/item?id=11141174. If you violate the rules of this community that badly, you forfeit the right to comment here.
We detached this comment from https://news.ycombinator.com/item?id=11141103 and marked it off-topic.
How do I vouch for a dead comment?