Index — Durpiness

Recoving old shitty DVD Backups

written by kousu, on Feb 19, 2012 9:33:00 PM.

Here's the OpenBSD 4.9 commands I'm using to tear through the 20+ DVDs of old backups from the era just after napster was killed and kazaa was dying, and before hundred-gig harddrives were common: Pop the disk, wait 10 seconds (so I can put the next one in), and snitch its volume label
umount /dev/cd0c && eject cd0 && sleep 10 && eject -t cd0 && mount -t udf /dev/cd0c /mnt/cd0 && cd-info --dvd | grep Volume | head -n 1
Not that I have kern.usermount=1 (equivalent to setting "user" in the options field in fstab, but more sane because it applies to all mounts instead of specific ones in fstab). Copy the data, using rsync because it gives progress and is restartable
sudo rsync -rvPh --chmod=D+X,F+rw /mnt/cd0/ /media/incoming/G/G_B_3/
Note that I change the G_x_x part depending on what cd-info tells me. Once I have everything copied on I'm going to run fdupes(1) (probably overnight), and then erase all the crap that I already had. See, I *know* some of these disks contain data I lost, but I also know a lot of them are full of things I have elsewhere still. I'm not reallllly sure how that happened, except that maybe I erased larger things that I'd backed up (I was running on a 40gig drive until 2nd year after all). rsync tells me it's running at 10.28MB/s. Copying a single disk is taking a pretty consistent 8 minutes. Curious, on a iso9660+Joliet filesystem, transfers are distinctly slower from a UDF one:
sent 4.71G bytes received 1.32K bytes 3.46M bytes/sec total size is 4.71G speedup is 1.00 time 22m41.921s

rsync vs find vs permissions

written by kousu, on Feb 19, 2012 8:33:00 PM.

UDF (the standard DVD format) is a headache. OpenBSD won't mount it by default, and even on Linux most UDF filesystems mount as 444 (that is, their root directory, not to mention every subdir, has -x, so you can't read anything unless you are root). I tried to use rsync --chmod, which isn't very well documented---and eventually found the magic combination: you need to say rsync -r --chmod=D+X,F+rw which means force +x on for all dirs and +rw on for all files from the sending side, ignoring what their original perms are. That X is capitalized seems to be key, though the manpage doesn't explain what the difference is (grr). I was going to write about how using

find -type d | while read line; do chmod +x "$line"; done

didn't work because find would generate the list of dirs before chmoding them, which was of course missing subdirs we didn't have a+xcess to, but using the (i guess more proper)

find -type d -exec chmod +x {} \;

seems to do the chmoding before trying to access the subfiles to see if they match the filter in the first, but that was just STUPID (though, nonobvious). The real problem here is that Unix overloaded the +x bit, and has in 40 years hasn't developed common tools for separating the two cases.

Automatic Reverse Tunnels with autossh

written by kousu, on Feb 9, 2012 8:33:00 PM.

Today I set up a reverse tunnel from one server of mine to another, that resets itself if it ever goes down, and I did it all remotely.

On a whim I turn one of the many scrap desktops my parents collect into an OpenBSD 5.0 server at their house, to have a spare backup digital presence for all the times. I used ddclient with zoneedit.com to name it greg.kousu.ca (especially important for their house, as they are on the local commodity ISP who rotates their IP regularly), which didn't take very long at all to setup, and configured their router to forward port 22 to 192.168.1.87:22. With ssh available, I left it alone, confident I could configure anything else remotely (with maybe the odd begging to my dad to please hit the power button).

Now, not trusting that their router is going to stay stable or that I'll always have access to it, I started dreaming about setting up a reverse tunnel--have greg connect home to kousu.ca such that kousu.ca can forward traffic into greg. This bypasses the NAT and firewall on the router, and it will always work so long as their house is online because it doesn't depend on telling the NAT beforehand which port to forward.

I couldn't believe it, as soon as I look on openports.se for an unrelated ssh program, I discovered autossh: http://openports.se/sysutils/autossh. It does exactly what it sounds like: it makes an SSH connection, and if the connection ever dies it remakes it.

After fiddling and figuring autossh out (which you can ask me about in the comments if you are stuck), I came up with this command line, which is to be run on greg:

autossh -M 22222 -vv -R 40404:greg:22 kousu.ca sh

This forwards for 40404 on kousu.ca (actually not on kousu.ca, on nest which is a machine sitting behind the NAT where kousu.ca is) to greg:22; this is the reverse part. A regular tunnel forwards a port on the machine you're coming from through to somewhere else, making it look like its coming from where you remote to. A reverse tunnel forwards a port from the remote machine back to some connection here. In this case, we're just connecting to the local machine, but it could read -R 40404:google.com:80 to forward connections to nest:40404 to google.com:80, via greg. As an aside, you can even have multiple tunnels at once, though the ssh manpage says nothing about this, just do something like 'ssh -R 8888:google.com:80 -R 7777:hotmail.com:80 kousu.ca -p 20202'.

The 'sh' on the end is just to give it something to launch that will hang around forever so the connection doesn't immediately die. The -M seemed to be necessary to make autossh do anything, though the manpage seemed to imply it was optional, but whatever. Also, pay attention: I didn't put 'greg.kousu.ca' in there, and I can do this because the tunnel comes *out* at greg, so the destination is resolved from greg's perspective (and greg knows it's greg).

Oh, you'll need to have ssh public-key authentication set up for whatever user you use to ssh in with. Otherwise, the daemon will be prompted for a password and probably just die since it's wouldn't be connected to a tty. If you don't know what public key auth is, look up ssh-keygen and ssh-copy-id.

I scraped together an rc.d script (yayyyy, OpenBSD finally saw the light and added these) to automate this at boot time,

$ cat /etc/rc.d/ssh_home_tunnel
#!/bin/sh

daemon="/usr/local/bin/autossh"
daemon_user=kousu #XXX insecure to use my personal account as the tunnel?
daemon_flags="-M 22222 -vv -R 40404:greg:22 kousu.ca sh"

. /etc/rc.d/rc.subr

rc_cmd $1

, rebooted, waited, and it worked!

To prove it worked, here's a test showing how long it goes from rebooting to being able to use the tunnel again:

nguenthe@nettop21:~$ ssh kousu@kousu.ca   #log into home
kousu@kousu.ca's password: 
Last login: Thu Feb  9 19:18:54 2012 from nettop21.student.cs.uwaterloo.ca
OpenBSD 4.9 (GENERIC.MP) #794: Wed Mar  2 07:19:02 MST 2011

Welcome to OpenBSD: The proactively secure Unix-like operating system.

Please use the sendbug(1) utility to report bugs in the system.
Before reporting a bug, please try to reproduce it with the latest
version of the code.  With bug reports, please try to ensure that
enough information to reproduce the problem is enclosed, and if a
known fix for it exists, include that as well.

~$ ssh localhost -p 40404   #in order to reboot greg, login
                            #via the tunnel from home.
                            #For the confused: the tunnel's
                            #mouth is at nest:40404, which
                            #is also localhost:40404 because
                            #we're on nest at the moment
Last login: Thu Feb  9 14:15:53 2012 from 192.168.1.1
OpenBSD 5.0 (GENERIC.MP) #59: Wed Aug 17 10:19:44 MDT 2011

DON'T EAT THE YELLOW CHEESE
$ sudo reboot               #reboot greg
Password:
Connection to localhost closed by remote host.
Connection to localhost closed.

###########
# The test!
~$ while true; do date; ssh localhost -p 40404;sleep 3; done
Thu Feb  9 20:22:59 EST 2012
ssh: connect to host localhost port 40404: Connection refused
Thu Feb  9 20:23:02 EST 2012
ssh: connect to host localhost port 40404: Connection refused
Thu Feb  9 20:23:05 EST 2012
ssh: connect to host localhost port 40404: Connection refused
Thu Feb  9 20:23:08 EST 2012
ssh: connect to host localhost port 40404: Connection refused
Thu Feb  9 20:23:11 EST 2012
ssh: connect to host localhost port 40404: Connection refused
Thu Feb  9 20:23:14 EST 2012
ssh: connect to host localhost port 40404: Connection refused
Thu Feb  9 20:23:17 EST 2012
ssh: connect to host localhost port 40404: Connection refused
Thu Feb  9 20:23:20 EST 2012
ssh: connect to host localhost port 40404: Connection refused
Thu Feb  9 20:23:23 EST 2012
ssh: connect to host localhost port 40404: Connection refused
Thu Feb  9 20:23:26 EST 2012
ssh: connect to host localhost port 40404: Connection refused
Thu Feb  9 20:23:29 EST 2012
ssh: connect to host localhost port 40404: Connection refused
Thu Feb  9 20:23:32 EST 2012
ssh: connect to host localhost port 40404: Connection refused
Thu Feb  9 20:23:35 EST 2012
ssh: connect to host localhost port 40404: Connection refused
Thu Feb  9 20:23:38 EST 2012
ssh: connect to host localhost port 40404: Connection refused
Thu Feb  9 20:23:41 EST 2012
ssh: connect to host localhost port 40404: Connection refused
Thu Feb  9 20:23:44 EST 2012
ssh: connect to host localhost port 40404: Connection refused
Thu Feb  9 20:23:47 EST 2012
ssh: connect to host localhost port 40404: Connection refused
Thu Feb  9 20:23:50 EST 2012
ssh: connect to host localhost port 40404: Connection refused
Thu Feb  9 20:23:53 EST 2012
ssh: connect to host localhost port 40404: Connection refused
Thu Feb  9 20:23:56 EST 2012
ssh: connect to host localhost port 40404: Connection refused
Thu Feb  9 20:23:59 EST 2012
ssh: connect to host localhost port 40404: Connection refused
Thu Feb  9 20:24:02 EST 2012
ssh: connect to host localhost port 40404: Connection refused
Thu Feb  9 20:24:06 EST 2012
Last login: Thu Feb  9 15:19:28 2012 from 192.168.1.1
OpenBSD 5.0 (GENERIC.MP) #59: Wed Aug 17 10:19:44 MDT 2011

DON'T EAT THE YELLOW CHEESE
$

Rollerblading

written by kousu, on Aug 4, 2011 2:36:39 AM.

Oh man, I am sitting on my bed and I feel like I'm still rollerblading. I'm rocking side to side just a bit here and I can feel the momentum just as if I was carving down an empty night street.

S&S

written by kousu, on Jul 21, 2011 7:32:00 PM.

[19:30] Sashbot: i'm a hopeless semantic
[19:30] Sashbot: what can i say

Direct

written by kousu, on Jul 20, 2011 11:09:00 PM.

This week's theme is genuiosity. I can't stand my usual tongue-in-cheek nothing-really-matters brou-ha-hah attitude. It makes me sick, and this feeling comes out in my musical tastes. I can't listen to say, most of Of Montreal or John Darnielle right now. Instead I've been listening to a lot of A Sunny Day In Glasgow and Elf Power, Lily Allen, ancient polyphony and plainchant (e.g. The Tallis Scholars) and this thing. Just good, plain honest quality music, with no sneering pretentions to it. Well that and Public Enemy. Everything else I have is too pounding for my mood or too fucking twisted and irritating. I've even discovered I want to listen to punk, which should say something.

Focusing too much on being honest and true and genuine is what I already do, and I've just fallen into traps of cynicism from that. I'm trying to break the cycle, being sick of yourself (okay, one nod to Of Montreal: "I spent a winter in Norway/Trying to restructure my character/Because it had become intolerable/to its author" or something) certainly helps. But Hm.

Scurrying Life

written by kousu, on Jul 11, 2011 9:38:00 PM.

I think it must suck to be an ant.

17

written by kousu, on Jul 9, 2011 5:14:00 AM.

17 is my favourite number. It is a prime, but it's not a really huge impossibly stupid prime for crypto that I--and you--can't wrap our heads around, and neither is it one of those "in" primes like 3, 2 or 5. It has a 7 in its base-ten representation which is a nice yellow number, but it's not actually a boring old "lucky" seven, it's SEVENTEEN.

Also they only want you when you're seventeen.

MPD Hackery

written by kousu, on Jul 9, 2011 4:44:00 AM.

Hackery is just so loverly. Even if it does mean I forget to nourish my cells.

So today I installed mpd finally on my shiny new server that this blog is on--and also where all my music is. I intended to set it up streaming to icecast but discovered that mpd actually comes with an http audio streaming server built in, which is trivial to turn on in /etc/mpd.conf:

audio_output {
	type		"httpd"
	name		"nest.kousu.ca mpd stream"
	port		"8888"
	quality		"3.0"
	format		"44100:16:1"
}

I actually fiddled with icecast a bit, but I couldn't make it fit nicely onto my server. With transmission I could hack the URLs in lighttpd+mod_proxy to make http://kousu.ca/transmission/web point at a separate webserver (the one that transmission-daemon runs internally) and still work right. Without icecast in the way (which is picky about the URLs it eats), I did manage to get this working so that http://kousu.ca/mpd is a live ogg vorbis stream of whatever is playing in my basement at this very moment. Here's the lighttpd.conf magic:

$HTTP["url"] =~ "^/mpd$" {
    proxy.server = (
      "" => (
        "host" => (
          "host" => "127.0.0.1",
          "port" => 8888
        )
      )
    )
}

Now, this system isn't as responsive as mounting samba shares and playing "locally", but it has the advantage of keeping the music player next to where the music actually is, which is really nice

I do want to push latency down in order to get it to respond quickly when I change tracks or track position in a client, so I lowered the encoder's quality from the default of 5 to 3.0 (and this has the advantage of using less bandwidth from my house). Another source of latency is in the listeners themselves, who will buffer a certain amount of data--a certain amount of music-time--before playing. I'm not going to be able to eliminate lag with this but I can probably get it reasonable and I'm too fiddly anyway--it's not like I need to be switching tracks every three seconds!

Another goal I have is to just basically have this work like original radio: I tune in some player somewhere and it just plays when there is a signal--it would be really annoying if you had to reset your radios everytime the station you're listening to goes off the air. Sadly, the weirdly semi-stateful HTTP mess that we've got makes exactly that happen. but eh, I've found a solution, and it is called "-loop 0":

$ mplayer -cache-min 3 -cache 500 -loop 0 http://kousu.ca/mpd

The -cache-min is the percentage of -cache when mplayer will start outputting sound, pushed low lower latency so that fiddling with sonata or whatever client I'm using, the music changes with only a second's delay or so. The -loop 0 means that when the stream dies mplayer tries to reload it until it comes back. My other plan was to see if I could a) hack mpd to or b) hack up some sort of proxy server that would c) insert blank sound (maybe even soothing gaussian noise) if there was no actual output. This is probably better bandwidth-wise, but it's kind of annoying to have a poll in a tight loop like that. Writing this now I realize that what this really calls for is d) UDP...and I wonder if there just might be a way to pull that off using mpd's pipe output--output to a pipe, encode, decode, and have a custom program reading ogg vorbis packets on the other end (and if the packets get out of order then whatever--the quirks of the digital future :P!).

Let's see how well it actually works:

MPlayer SVN-r33713-4.6.0 (C) 2000-2011 MPlayer Team
163 audio & 362 video codecs
mplayer: could not connect to socket
mplayer: No such file or directory
Failed to open LIRC support. You will not be able to use your remote control.

Playing http://kousu.ca/mpd.
Resolving kousu.ca for AF_INET6...

Couldn't resolve name for AF_INET6: kousu.ca
Resolving kousu.ca for AF_INET...
Connecting to server kousu.ca[76.10.149.133]: 80...

Cache size set to 500 KBytes
Cache fill:  2.36% (12102 bytes)   

[Ogg] stream 0: audio (Vorbis), -aid 0
Ogg file format detected.
Clip info:
 Artist: Shpongle
 Title: Shpongleyes
 Album: Tales of the Inexpressible
 Creation Date: 2001
 Genre: Ambient Trance
Ogg : bad packet in stream 0
==========================================================================
Opening audio decoder: [ffmpeg] FFmpeg/libavcodec audio decoders
[vorbis @ 0x8a97820]partition out of bounds: type, begin, end, size, blocksize: 1, 0, 112, 16, 1024
[vorbis @ 0x8a97820] Vorbis setup header packet corrupt (residues). 
[vorbis @ 0x8a97820]Setup header corrupt.
Could not open codec.
ADecoder init failed :(
ADecoder init failed :(
Requested audio codec family [vorbis] (afm=libvorbis) not available.
Enable it at compilation.
Opening audio decoder: [tremor] Ogg/Vorbis audio decoder
AUDIO: 44100 Hz, 1 ch, s16le, 80.0 kbit/11.34% (ratio: 10000->88200)
Selected audio codec: [tremor] afm: tremor (OggVorbis audio)
==========================================================================
[AO OSS] audio_setup: Can't open audio device /dev/dsp: No such file or directory
AO: [alsa] 44100Hz 1ch s16le (2 bytes per sample)
Video: no video
Starting playback...
A: 429.7 (07:09.6) of 0.0 (unknown)  0.4% 1% 
Cache not filling, consider increasing -cache and/or -cache-min!
A: 430.1 (07:10.1) of 0.0 (unknown) 31.9% 0% 
Cache not filling, consider increasing -cache and/or -cache-min!
A: 442.7 (07:22.7) of 0.0 (unknown) 15.0% 0% 
Cache not filling, consider increasing -cache and/or -cache-min!

Hmm. Maybe I pushed the latency a bit too low. But it isn't skipping in listening to it. I can stop mpd playing and restart it. I wonder if it would be as resiliant if I wasn't on ethernet, though. I guess I'll find out at work on Monday!

p.s. Sphongle sucks

Porxii

written by kousu, on Jul 5, 2011 1:11:00 AM.

Definition: Porxis: See Porcoplaxis
. Pl. Porxii. Definition: Porcoplaxis: The particular nugget of a personality that guides a life to ruin. Usually in Compare: Modus Operandi, Idée fixe.

Archive