And are you reading that visually in Morse code? I would have thought a different/more optimized flashing pattern that required a smartphone to read would be faster, especially if we're talking IPv6 addresses, but that's just me.
Yes. I made it flash 100ms-on-900ms-off for a ZERO, and 500ms-on-500ms-off for a ONE. There's a 1000ms-on-1000ms-off flash at the beginning to indicate the start point. And it's IPv4, so there's only 32 bits.
So it's slow enough to read out and type out into a text editor, then convert the typeout to its IP, but not slow enough that it's a bother.
To show the 10 most recent firewall log entries, the dashboard runs `clog filter.log | tail -10r` (`tail -10r` is the equivalent of `tail -10 | tac`). It would be nice to instead just run `clog -f filter.log` (open filter.log in follow mode, just like `tail -f`), leave it open across dashboard refresh iterations, and maintain an in-memory buffer of the 10 most recent entries for display. It would presumably be more efficient.
However this requires the ability to do a non-blocking read of the command output, which requires `PROCINFO["read_timeout"]`, which is gawk-specific and not available on FreeBSD's default awk.
I don't mind it too much, because repeatedly doing `clog -f | tail -10r` doesn't use that much CPU to be concerned about anyway. Though I may investigate Perl to see if it's possible there.
while (1) {
try_reading_one_more_line( $file_in );
print_last_10_lines_reverse();
sleep( 1 ); # wait, in this case one second
}
and that doing on the opened file $file_in. You'd have to write both functions in the loop of course. But Perl is a real language which allows you to try reading from the point in the file you've reached the last time, without reading anything you've already read again. POSIX semantics allows you to observe that the file grew even if in the previous pass you've hit the end of file. And you even don't have to keep in memory the lines which aren't to be printed anymore.
I prefer Perl to any shell scripts, as soon as the script is more than circa 10 lines.
>But Perl is a real language which allows you to try reading from the point in the file you've reached the last time, without reading anything you've already read again.
The input in this case is the stdout of a command (clog), so it's not a seekable file.
The flow I've described works on non-seekable files (and pipes). After you reach the end of file in one pass, you are allowed to try to read again in the next, and the file pointer stayed on the same position if you haven't closed the file (and of course you should not close it). If the other process wrote something, the next time read will succeed. If you don't want to display the partial line before the line feed is written, you have to collect the read bytes before you read the line feed. It's all doable in Perl.
The only case I've seen something of what I've described not working was some case under some non-recent version of Wine, and I don't know if they fixed that. Otherwise Perl and POSIX support everything described.
Okay, I thought you were trying to say that perl could resume reading from the last position of the file after closing and reopening it, which would require a seekable file to work.
If you were referring to leaving the file open across multiple iterations, then yes that is already what I said in the first post that you responded to. It's obvious that perl can do it; any program can.
In awk, `"foo" | getline` spawns `foo` and reads the first line from it, or reads the next line from the previous invocation of `foo` if that hasn't been closed yet. You have to explicitly call `close("foo")` to make the next invocation of `"foo" | getline` spawn a new `foo` instead of reading another line from the previous one.
In shell, you'd just move the invocation of `foo` outside the dashboard loop, ie `foo | while :; do if read -r line; then ...; fi; sleep 1; done`
The hard part is not to read from the same command across multiple dashboard iterations. The hard part is detecting that there is nothing more to read from the command without blocking, so that the dashboard can continue rendering if there's no more input. As I wrote in the first post that you responded to, gawk has `PROCINFO["read_timeout"]` to help with this (`getline` will return a specific error that the script can detect and break out of the loop), but FreeBSD's awk does not, which is why I was thinking of investigating whether Perl can do it.
I'd have gone for "can't use perl, that's obsolete (uses awk instead)" which is kind of ironic if you remember the days of a2p. As it is, a combination of sh/awk will do the job, but is likely slower and less flexible than doing it in perl because this is exactly the sort of job perl was designed for, back in the day.
The code is what I described in https://news.ycombinator.com/item?id=21567202 . It's equivalent to first converting 0 to E and 1 to T, then transmitting that. It's not using the representations of 0 and 1 themselves, which are five times longer.
> The default FreeBSD shell is tcsh. I was prepared to not have bash, but tcsh is quite alien in its syntax compared to regular POSIX sh. I decided to ignore it and just use POSIX sh.
Would FreeBSD consider POSIX sh as their default shell instead of tcsh?
You can easily change your login shell with chsh. If you install bash first, you can change your shell to bash. It's an inconvenience to change it, but changing the default is a major effort.
POSIX sh is a sensible default; and the technical work to do it would take less time than we've spent writing these comments. However, actually changing the default, would be a major effort; you'd need to form a consensus that the new default is better than the old default, that it's so much better than the old default that invalidating any documentation that relies on the old default is worth it.
Note that if you do this, you should either `pkg install bash-static` or build bash from the ports using STATIC option (I don't _believe_ it's the default, though it really should be). Without that static option, you could potentially break your shell by upgrading related packages.
Though one would generally write a shell script starting with:
`#!/bin/sh`
So it doesn't really matter what the login shell is. (As a long-time FreeBSD user, the first thing I always do is install a different shell; I'm not sure who actually likes tcsh.)
Ah, I didn't know the default for new users was different. Everything I did for the dashboard was as the root user (some things the dashboard needs to read are ACLed to root). I'll fix the post.
> The default FreeBSD shell is tcsh. I was prepared to not have bash, but tcsh is quite alien in its syntax compared to regular POSIX sh.
All FreeBSD scripts I encountered use posix shell.
> does have a dependency on perl to get the current time in seconds from the Unix epoch. This is because FreeBSD's date does not have a way to get milliseconds in the time
One could just install the GNU coreutils (probably, not checked though, much less overhead than a full perl install) or kill that problem with very few lines of C.
Yes, I would've done that for a regular FreeBSD install. But as I wrote a little before that part, because it was a pfSense install, I decided not to install any additional `pkg` packages myself.
> All FreeBSD scripts I encountered use posix shell.
Defaults matter, and it's helpful when the default shell for command-line use matches the shell used for scripting, because people often do scripting on the command line.
/bin/sh is mostly intended for scripts, less so as an interactive user shell. It needs to be small, fast and as bugfree as possible, as opposed to offering a plethora of user interactive features. That's why FreeBSD uses sh and debian uses dash for this purpose.
I use Debian, and I'm aware. Bash still uses primarily POSIX sh syntax, with extensions. It's helpful for people to be able to apply knowledge from the command line to shell scripting and vice versa, even if they have to do a bit of porting work, rather than using completely different shells and syntax.
The default shell (what "adduser" offers as the default login shell) in FreeBSD is sh (/bin/sh). The root's shell is /bin/csh given FreeBSD's BSD roots. Though pfsense may have changed its defaults compared to a stock FreeBSD release.
(duplicating from other comments) Bash is indeed not in FreeBSD base; it's easy to install it, but not installing it is also a valid choice. The default login shell doesn't make a big difference, you can use chsh to change that. I also agree POSIX sh isn't quite enough for a lot of scripts.
> [ sysfs, procfs]
These are sort of available, but really just for compatibility with Linux binaries; take a look at the man pages for linsysfs and linprocfs. Just a note to say they're not totally missing, but I would agree they probably don't contain what you're looking for anyway.
> In some cases the sysctl output is not easily machine-parseable. For example, the uptime information from sysctl -n kern.boottime
Depending on how easy it is to process binary values, sysctl -b kern.boottime could be useful here; you'd get two raw integers.
> FreeBSD's netstat -I em0 -bin returns a tabular display,
You might try netstat -I re0 -bin --libxo json or similar. libxo was added in FreeBSD 11.0, and I think got plumbed to the networking commands at least (I see it's not hooked to sysctl in 12.0, so probably not in whatever 11.X pfSense is on either, although it would be useful). check man xo_parse_args for more details.
> nothing recognizes --help
--help is a GNUism; usually the BSD version is -h (although, not for netstat), and yeah, the included help text isn't super useful unless you know what the things mean... pfSense is intended for resource constrained systems, which I guess justifies no man text... but I'd install it it's possible, it's super useful.
> Perl would probably be another good choice
(this isn't accurate, as noted by comments) Perl is a nice choice on FreeBSD, because it is part of the base system (unlike, as you noted, bash). This can sometimes be a bit tricky, if you want a new perl feature, but as long as you're writing run of the mill perl (which everything described here should fit), as long as it's not a positively ancient system, you should be ok.
Anyway; it looks like you've learned quickly; enjoy!
`sysctl -b kern.boottime` works wonderfully. The output is two 64-bit integers, so the script pipes it through `od -t uI`. I would've liked to use `-t uL` but `od` complains that it's a "bad byte count", so the script has to manually add the two components together. (`-t dL` also raises the same error, even though the man page implies that at least should work. Strange. I did not investigate why.)
It also works for the thermal sysctls - they appear to output 4-byte unsigned integers in deci-Kelvins. Eg 3061 is 306.1 K == 33.0 degrees Celsius.
`netstat ... --libxo json` does work, though parsing JSON from awk isn't trivial either. The best way I can think of is to use `json,pretty` which puts one key-value-pair per line, which I could then filter for the specific keys I want. But it would amount to as much or more effort as the non-JSON parser makes.
pfSense doesn't have `jq` by default but does have `uclcmd`. I got as far as `netstat -I em0 -bin --libxo json | uclcmd get -f - -j '.statistics.interface'` to get an array of JSON objects, but any attempt to apply a transformation to each element of this array (there are no manpages for it online, but its Github readme implies the existence of an `each` function) caused the program to segfault.
In some cases the sysctl output is not easily machine-parseable. For example, the uptime information from sysctl -n kern.boottime
This largely misses the point. The output of sysctl(8) is not intended to be machine readable. Instead of parsing strings you should be using sysctl(3) directly or through whatever bindings for your language of choice.
I don't believe POSIX shell or awk have bindings for sysctl(3)?
That's essentially what sysctl(8) is. Unlike Linux though the text interface isn't really intended to be the primary API. The article discusses using Rust and C which both have access to sysctl(3).
Perl hasn't been part of the base system since FreeBSD 8 or so. It's also fairly recent, 5.26 I think, whereas the current one is 5.30 (even minor versions numbers are stable).
Also pfSense is a constrained environment wothout all the feature of a full FreeBSD install.
It's definitely there on pfSense 2.4.4 (based on FreeBSD 11.2), which is why I mentioned it in the blog post. So if it isn't part of FreeBSD base then pfSense is installing it anyway.
> --help is a GNUism; usually the BSD version is -h (although, not for netstat), and yeah, the included help text isn't super useful unless you know what the things mean...
OTOH, FreeBSD's man pages[0] are superb compared to that of Linux for most subjects. I usually don't bother with -h and go straight to the man pages.
I love that you can comment by sending an email with the title of the post in the subject.
>You can post comments on this blog post by: sending mail to email@address.dev with a subject starting with [blog-2019-10-20-local-linux-user-tries-freebsd]
I used to use Freebsd. It was great until it wasn't. Mainly, I would recommend not using it as a desktop. It's perfect for a server or FreeNAS but the ports for common desktop applications seem to be broken fairly often which causes usability issues and some very heavy frustration. From the forums (you will use them allot), I learned that the developers mainly use Macs as their primary PC's. So this problem is not likely to change.
As far as Awk over Perl. It's been quite some time but when I did awk scripts they had some pretty annoying quirks that you just had to learn. Perl was just easier. Has that changed? I know that Perl as a larger language is languishing but isn't it still perfect for this type of thing?
> It's perfect for a server or FreeNAS but the ports for common desktop applications seem to be broken fairly often which causes usability issues and some very heavy frustration.
I briefly tried FreeBSD. Only briefly, because that ports thing was a mess. My trial lasted only a few days before my system was an unusable mess.
Part of that is, it's just a fairly steep learning curve. I did learn to stick with one way of doing ports. I used Portmaster then you learn not to troubleshoot too much as you can really make a mess of things. You need to let the tools work for you, but again... when the ports themselves are broken (i.e. not your machine/dependency .."heck") then you can't do anything.
For the 'best of both worlds' scenario, use Slackware Linux in place of FreeBSD.
Slackware is one of the very oldest Linuxes, and traditionally used a BSD-flavored 'init' system, rather than a System-V syle init, or even the newer 'systemd'. It has traditionally concentrated on being extremely stable, and more slanted towards server work, though I used it for many years as my main desktop.
I would say it's more slanted towards desktop use than server use.
Generally you want to automate server management with some sort of config management. While possible with Slackware, it's easier with a Debian or Red Hat style system.
For a desktop Slackware is pretty nice. It's easy to understand and gets out of your way. I used it for over a decade until, one day, I realised I was tired of recompiling slackbuilds after each low level dependency upgrade.
75 comments
[ 3.2 ms ] story [ 156 ms ] threadI wish my day job was this exciting!
So it's slow enough to read out and type out into a text editor, then convert the typeout to its IP, but not slow enough that it's a bother.
I typically end up just nmapping :-/
* https://www.freebsd.org/cgi/man.cgi?query=led&sektion=4
* https://www.freebsd.org/cgi/man.cgi?query=morse&sektion=6
To show the 10 most recent firewall log entries, the dashboard runs `clog filter.log | tail -10r` (`tail -10r` is the equivalent of `tail -10 | tac`). It would be nice to instead just run `clog -f filter.log` (open filter.log in follow mode, just like `tail -f`), leave it open across dashboard refresh iterations, and maintain an in-memory buffer of the 10 most recent entries for display. It would presumably be more efficient.
However this requires the ability to do a non-blocking read of the command output, which requires `PROCINFO["read_timeout"]`, which is gawk-specific and not available on FreeBSD's default awk.
I don't mind it too much, because repeatedly doing `clog -f | tail -10r` doesn't use that much CPU to be concerned about anyway. Though I may investigate Perl to see if it's possible there.
I prefer Perl to any shell scripts, as soon as the script is more than circa 10 lines.
Yes.
>But Perl is a real language which allows you to try reading from the point in the file you've reached the last time, without reading anything you've already read again.
The input in this case is the stdout of a command (clog), so it's not a seekable file.
The flow I've described works on non-seekable files (and pipes). After you reach the end of file in one pass, you are allowed to try to read again in the next, and the file pointer stayed on the same position if you haven't closed the file (and of course you should not close it). If the other process wrote something, the next time read will succeed. If you don't want to display the partial line before the line feed is written, you have to collect the read bytes before you read the line feed. It's all doable in Perl.
The only case I've seen something of what I've described not working was some case under some non-recent version of Wine, and I don't know if they fixed that. Otherwise Perl and POSIX support everything described.
If you were referring to leaving the file open across multiple iterations, then yes that is already what I said in the first post that you responded to. It's obvious that perl can do it; any program can.
I'm not aware how shell scripts or awk could do that. Maybe you can explain. I knew about Perl and that's what I showed.
In shell, you'd just move the invocation of `foo` outside the dashboard loop, ie `foo | while :; do if read -r line; then ...; fi; sleep 1; done`
The hard part is not to read from the same command across multiple dashboard iterations. The hard part is detecting that there is nothing more to read from the command without blocking, so that the dashboard can continue rendering if there's no more input. As I wrote in the first post that you responded to, gawk has `PROCINFO["read_timeout"]` to help with this (`getline` will return a specific error that the script can detect and break out of the loop), but FreeBSD's awk does not, which is why I was thinking of investigating whether Perl can do it.
In Perl $file->blocking( 0 ) or fcntl with O_NONBLOCK enables non-blocking sysread, I’ve recently used that.
https://perldoc.perl.org/IO/Handle.html
https://perldoc.perl.org/functions/fcntl.html
https://perldoc.perl.org/functions/sysread.html
Wouldn't it be a lot easier to use mdns/zeroconf, or static IPs? Or use a label maker and put the MAC address on it, and then use arp-scan?
Would FreeBSD consider POSIX sh as their default shell instead of tcsh?
`#!/bin/sh`
So it doesn't really matter what the login shell is. (As a long-time FreeBSD user, the first thing I always do is install a different shell; I'm not sure who actually likes tcsh.)
All FreeBSD scripts I encountered use posix shell.
> does have a dependency on perl to get the current time in seconds from the Unix epoch. This is because FreeBSD's date does not have a way to get milliseconds in the time
One could just install the GNU coreutils (probably, not checked though, much less overhead than a full perl install) or kill that problem with very few lines of C.
Defaults matter, and it's helpful when the default shell for command-line use matches the shell used for scripting, because people often do scripting on the command line.
If you need more-than-POSIX in your scripts, then use the proper shebang.
!man netstat
> shell things
(duplicating from other comments) Bash is indeed not in FreeBSD base; it's easy to install it, but not installing it is also a valid choice. The default login shell doesn't make a big difference, you can use chsh to change that. I also agree POSIX sh isn't quite enough for a lot of scripts.
> [ sysfs, procfs]
These are sort of available, but really just for compatibility with Linux binaries; take a look at the man pages for linsysfs and linprocfs. Just a note to say they're not totally missing, but I would agree they probably don't contain what you're looking for anyway.
> In some cases the sysctl output is not easily machine-parseable. For example, the uptime information from sysctl -n kern.boottime
Depending on how easy it is to process binary values, sysctl -b kern.boottime could be useful here; you'd get two raw integers.
> FreeBSD's netstat -I em0 -bin returns a tabular display,
You might try netstat -I re0 -bin --libxo json or similar. libxo was added in FreeBSD 11.0, and I think got plumbed to the networking commands at least (I see it's not hooked to sysctl in 12.0, so probably not in whatever 11.X pfSense is on either, although it would be useful). check man xo_parse_args for more details.
> nothing recognizes --help
--help is a GNUism; usually the BSD version is -h (although, not for netstat), and yeah, the included help text isn't super useful unless you know what the things mean... pfSense is intended for resource constrained systems, which I guess justifies no man text... but I'd install it it's possible, it's super useful.
> Perl would probably be another good choice
(this isn't accurate, as noted by comments) Perl is a nice choice on FreeBSD, because it is part of the base system (unlike, as you noted, bash). This can sometimes be a bit tricky, if you want a new perl feature, but as long as you're writing run of the mill perl (which everything described here should fit), as long as it's not a positively ancient system, you should be ok.
Anyway; it looks like you've learned quickly; enjoy!
It also works for the thermal sysctls - they appear to output 4-byte unsigned integers in deci-Kelvins. Eg 3061 is 306.1 K == 33.0 degrees Celsius.
`netstat ... --libxo json` does work, though parsing JSON from awk isn't trivial either. The best way I can think of is to use `json,pretty` which puts one key-value-pair per line, which I could then filter for the specific keys I want. But it would amount to as much or more effort as the non-JSON parser makes.
pfSense doesn't have `jq` by default but does have `uclcmd`. I got as far as `netstat -I em0 -bin --libxo json | uclcmd get -f - -j '.statistics.interface'` to get an array of JSON objects, but any attempt to apply a transformation to each element of this array (there are no manpages for it online, but its Github readme implies the existence of an `each` function) caused the program to segfault.
This largely misses the point. The output of sysctl(8) is not intended to be machine readable. Instead of parsing strings you should be using sysctl(3) directly or through whatever bindings for your language of choice.
That's essentially what sysctl(8) is. Unlike Linux though the text interface isn't really intended to be the primary API. The article discusses using Rust and C which both have access to sysctl(3).
Also pfSense is a constrained environment wothout all the feature of a full FreeBSD install.
> --help is a GNUism; usually the BSD version is -h (although, not for netstat), and yeah, the included help text isn't super useful unless you know what the things mean...
OTOH, FreeBSD's man pages[0] are superb compared to that of Linux for most subjects. I usually don't bother with -h and go straight to the man pages.
[0] https://www.freebsd.org/cgi/man.cgi
>You can post comments on this blog post by: sending mail to email@address.dev with a subject starting with [blog-2019-10-20-local-linux-user-tries-freebsd]
You can start with popen to some command, then when that's slow, switch to ffi calls.
As far as Awk over Perl. It's been quite some time but when I did awk scripts they had some pretty annoying quirks that you just had to learn. Perl was just easier. Has that changed? I know that Perl as a larger language is languishing but isn't it still perfect for this type of thing?
I briefly tried FreeBSD. Only briefly, because that ports thing was a mess. My trial lasted only a few days before my system was an unusable mess.
In what way was it a mess?
Slackware is one of the very oldest Linuxes, and traditionally used a BSD-flavored 'init' system, rather than a System-V syle init, or even the newer 'systemd'. It has traditionally concentrated on being extremely stable, and more slanted towards server work, though I used it for many years as my main desktop.
Generally you want to automate server management with some sort of config management. While possible with Slackware, it's easier with a Debian or Red Hat style system.
For a desktop Slackware is pretty nice. It's easy to understand and gets out of your way. I used it for over a decade until, one day, I realised I was tired of recompiling slackbuilds after each low level dependency upgrade.
Local Linux User Tries FreeBSD - You Won't Believe What Happens Next!
(Actually nothing much happened at all. A few differences, but nothing major.)