I Drivel » linux http://blog.justinlintz.com whatever comes to mind Wed, 12 Oct 2011 01:37:13 +0000 en hourly 1 http://wordpress.org/?v=3.2.1 NRPE returning no output? http://blog.justinlintz.com/2011/10/nrpe-returning-no-output/ http://blog.justinlintz.com/2011/10/nrpe-returning-no-output/#comments Wed, 12 Oct 2011 01:37:13 +0000 justin http://blog.justinlintz.com/?p=200 command[check_recent_core]=PLUGINPATH/check_recent_core.sh --file="$ARG1" --freshness=$ARG2$

Spot the error? I only wasted an hour of my life and another 30 minutes of co-workers trying to figure out why I kept getting a "NRPE: No output returned from plugin" error in Nagios. The issue? $ARG1 is missing a closing “$”. *slams head on desk*

]]> command[check_recent_core]=PLUGINPATH/check_recent_core.sh --file="$ARG1" --freshness=$ARG2$

Spot the error? I only wasted an hour of my life and another 30 minutes of co-workers trying to figure out why I kept getting a "NRPE: No output returned from plugin" error in Nagios. The issue? $ARG1 is missing a closing “$”. *slams head on desk*

]]> http://blog.justinlintz.com/2011/10/nrpe-returning-no-output/feed/ 0 Determining my most used focal length in photos with python and dcraw http://blog.justinlintz.com/2010/12/determining-my-most-used-focal-length-in-photos-with-python-and-dcraw/ http://blog.justinlintz.com/2010/12/determining-my-most-used-focal-length-in-photos-with-python-and-dcraw/#comments Tue, 28 Dec 2010 05:52:50 +0000 justin http://blog.justinlintz.com/?p=138 For the holidays I got a new Canon 7D camera. I’ve been saving quite a few amazon gift cards over the last year that I was previously planning on using towards purchasing the camera but thankfully Santa delivered me one. My current lens collection consists of

10-22mm f/3.5-4.5 70-200mm f/4 50mm f/1.8 18-55mm . . . → Read More: Determining my most used focal length in photos with python and dcraw]]> For the holidays I got a new Canon 7D camera. I’ve been saving quite a few amazon gift cards over the last year that I was previously planning on using towards purchasing the camera but thankfully Santa delivered me one. My current lens collection consists of

The lens collection has been built over the last 4 years from when I first got my Canon Rebel XT and I’ve been fairly happy with them thus far. As part of looking into what new lens to get I wanted to know what focal length I shot with the most over the previous years. A quick look with

find . -type f -name '*.CR2' | wc -l

shows I have 4697 RAW images in my photos folder, so I’ll need some sort of automated solution to figure this out.

My first inclination was to poke around Python’s PIL and see if it supported reading of RAW images, it didn’t appear to do so that option was out. I looked at the Canon RAW format at http://www.sno.phy.queensu.ca/~phil/exiftool/canon_raw.html and contemplated doing some parsing of the files to extract the info I needed from the EXIF data. I remembered in the past I had used UFRaw to play around with RAW conversion in Linux a while back and saw they are using a nice simple tool to do their RAW parsing called dcraw. It’s a small C program that I was able to compile the source with the following commands…

sudo apt-get install libjpeg-dev liblcms-dev
gcc -o dcraw -O4 dcraw.c -lm -ljpeg -llcms

Using dcraw seemed like the path of least resistance and now I just needed to wrap some code around it to go through all of my RAW images and extract the EXIF info and keep a hash table of the focal length value. I whipped up a small python script and posted a gist of it over on github.

It’s fairly straight forward and could easily be modified to do more, like collect aperture sizes or any other EXIF data. I may go back and build it out a bit more to make it more flexible in what data it collects, but for now this suited my needs at the time. My top 10 focal lengths ended up being

Total pics to process: 4697
Length: 50.0  , Count: 1624
Length: 22.0  , Count: 576
Length: 18.0  , Count: 423
Length: 200.0  , Count: 404
Length: 55.0  , Count: 373
Length: 10.0  , Count: 192
Length: 70.0  , Count: 176
Length: 37.0  , Count: 74
Length: 16.0  , Count: 33

Takes close to 11 seconds on average to calculate that for 4697 images.

I had been wanting to upgrade my 50mm f/1.8 lens for a bit as I’d really started to notice the short comings of it. The lens has a hard time focusing and also had a bit of a back focusing issue. Since my Uncle borrows my camera a lot when he goes on vacation I worked out a deal with him to purchase the 50mm f/1.4 for me and in exchange I’d give him my old Canon Rebel XT with the kit lens and the 50mm f/1.8 lens.

The kit lens was a great walk about lens so I’m currently trying to decide on a good and reasonably priced lens to cover somewhere in the 18-100mm range. I’m reading through a bunch of comments on amazon reviews and dpreview forums to figure out what my best bang for my buck would be for a lens in that range.

]]> http://blog.justinlintz.com/2010/12/determining-my-most-used-focal-length-in-photos-with-python-and-dcraw/feed/ 2 Python pip on Ubuntu Lucid Lynx http://blog.justinlintz.com/2010/09/python-pip-on-ubuntu-lucid-lynx/ http://blog.justinlintz.com/2010/09/python-pip-on-ubuntu-lucid-lynx/#comments Sun, 26 Sep 2010 04:48:24 +0000 justin http://blog.justinlintz.com/?p=135 Lately I’ve been reading up on Django a bit in an effort to get back into programming.  It seems the preferred way of running Django is inside a virtualenv environment.  I try to stick to packages when possible for installing applications and saw that the python-virtualenv package wasn’t too far behind at version 1.4.5, . . . → Read More: Python pip on Ubuntu Lucid Lynx]]> Lately I’ve been reading up on Django a bit in an effort to get back into programming.  It seems the preferred way of running Django is inside a virtualenv environment.  I try to stick to packages when possible for installing applications and saw that the python-virtualenv package wasn’t too far behind at version 1.4.5, where the latest is at 1.5.1.  This appeared at first to suit me just fine until I discovered the version of pip that gets installed along with python-virtualenv is at 0.3.1, where the latest is at 0.8.1.  The version of pip installed is so old that it doesn’t have support for the “uninstall” command yet, blech.  Luckily you can do the following from within pip to upgrade itself

sudo pip install --upgrade pip

Now back to reading about setting up a virtualenv environment

]]> http://blog.justinlintz.com/2010/09/python-pip-on-ubuntu-lucid-lynx/feed/ 1 Configuring Varnish http://blog.justinlintz.com/2010/08/configuring-varnish/ http://blog.justinlintz.com/2010/08/configuring-varnish/#comments Sat, 14 Aug 2010 03:22:19 +0000 justin http://blog.justinlintz.com/?p=103 At $WORK I’m currently working on deploying a pool of Varnish servers to sit in front of some Apache servers running Pressflow. On our current infrastructure we’ve been running Squid for the past few years with very good success , minus a hiccup or two along the way, one involving memory fragmentation (thank you . . . → Read More: Configuring Varnish]]> At $WORK I’m currently working on deploying a pool of Varnish servers to sit in front of some Apache servers running Pressflow. On our current infrastructure we’ve been running Squid for the past few years with very good success , minus a hiccup or two along the way, one involving memory fragmentation (thank you tcmalloc). Varnish has a few nice features that Squid lacks.

  • The ability to PURGE objects using wildcards
  • Better support for multiple processors (Squid can benefit from multi procs when using AUFS)
  • Grace period that can be configured to serve objects from the cache after they’ve expired while fetching the new content from the backend. You can also use this to serve up stale content if your backend is down
  • Ships with a nice set of command line tools (varnishtop,varnishlog,varnishstat,varnishhist,etc…)
  • A very flexible scripting/configuration language (you can even do inline C if you’re feeling saucy) that allows you to manipulate the objects at any point in the request or response (See flow chart)

    • There are many others but these are just a few off the top of my head and I’m still discovering what other capabilities Varnish has. The site has not gone live yet so I’m still testing on a dev version of the site and have not had an opportunity to perform any load testing yet. So far with my current working configuration I’ve made the following tweaks

  • Stripped cookies off static objects
  • Stripped Google analytics cookies
  • Removed empty cookies
  • Configured a graceful period to serve up stale objects from cache
  • Added a debugging header to show weather the object was a cache HIT or MISS

    • The use of mod_expires on the Apache backend controls cache times for static assets (css,js,images,etc..). In my googling around when reading about Varnish I see a lot of people are setting cache times in their VCLs. IMO you should be letting the backend or application itself control the TTLs on objects. Within your application you can set more defined TTLs for certain sections of your site or even certain types of dynamic content without having to rely on complex VCL rules or deal with the deployment of the rules into Varnish. While Varnish does support a “graceful” style restart, its not quite as eloquant as doing service apache graceful. Kristian Lyngstol (one of the Varnish devs) has a good post on his blog on dealing with this. Also with the use of mod_expires you can set TTLs based on MIME-type within Apache.

    One other thing I see a lot of people blindly recommending in configurations to deal with Varnish’s behavior of not caching cookies is to take the cookie value and add it into Varnish’s hash of the object. e.g.

    sub vcl_hash {
    set req.hash += req.http.cookie;
    }

    If a light bulb just went off in your head as to why this is a bad idea, kudos to you. What you’re basically doing is creating a cache per-user on your Varnish server. Your hit ratio will plummet from this config. There are scenarios where this can be used in a good way. In talking with some folks in #varnish on irc.linpro.no, a scenario where you’d want this is if say you had a cookie that was a display filter on your site or some sort of site customization that didn’t have a large number of combinations.

    One thing that bothers me about Varnish currently is that it’s admin interface is completely unsecured. By default it listens on localhost but without any authentication, anyone with a shell on your Varnish box can bring down your Varnish instance or modify the config in anyway they feel fit. For those that allow dev’s on production servers to debug logs, this is a bit of a security concern. I’m not really sure of a workaround for this, so if anyone has any ideas, leave it in the comments below.

    If you use Cacti for trending, there are some great templates available over at the cacti forums. They utilize a python script that needs access to the admin interface.

    I’ll probably post some more in the future on Varnish as I do further reading and testing with it.

    ]]>     http://blog.justinlintz.com/2010/08/configuring-varnish/feed/ 1     Thinkpad Trackpoint sensitivity on Ubuntu http://blog.justinlintz.com/2010/06/thinkpad-trackpoint-sensitivity-on-ubuntu/ http://blog.justinlintz.com/2010/06/thinkpad-trackpoint-sensitivity-on-ubuntu/#comments Sat, 05 Jun 2010 00:27:54 +0000 justin http://blog.justinlintz.com/?p=97 A while back I found some notes on configuring the sensitivity of my trackpoint on my Thinkpad T43 and took the time to tweak the values to get it just right. The commands were

    /bin/echo -n 171 > /sys/devices/platform/i8042/serio1/serio2/sensitivity /bin/echo -n 119 > /sys/devices/platform/i8042/serio1/serio2/speed

    In order to keep those values the same on reboot, . . . → Read More: Thinkpad Trackpoint sensitivity on Ubuntu]]> A while back I found some notes on configuring the sensitivity of my trackpoint on my Thinkpad T43 and took the time to tweak the values to get it just right. The commands were

    /bin/echo -n 171 > /sys/devices/platform/i8042/serio1/serio2/sensitivity
    /bin/echo -n 119 > /sys/devices/platform/i8042/serio1/serio2/speed

    In order to keep those values the same on reboot, I placed those commands in /etc/rc.local. I rebooted and… values got reset. After struggling a bit and just giving up on the issue, I ended up just making a shell script that I would execute on boot each time (crappy solution). Finally I got annoyed with the issue enough and researched it again some more and stumbled uponthis post on Ubuntu Forums. A couple of things I learned from that…

    1. The proper ways to set the values in the sysfs is to use udev rules, Writing udev rules
    2. Even if I used a udev rule, there appears to be an issue where the device does not exist yet at the time the rule is processed to set the value for the device

    Solution (per forum post) create a file /etc/udev/rules.d/trackpoint.rules and place the following in

    SUBSYSTEM=="serio", DRIVERS=="psmouse", ATTR{description}=="Synaptics pass-through", WAIT_FOR="/sys/devices/platform/i8042/serio1/serio2/sensitivity" ATTR{sensitivity}="171", ATTR{speed}="119"
    ]]> http://blog.justinlintz.com/2010/06/thinkpad-trackpoint-sensitivity-on-ubuntu/feed/ 4     Check for missing reverse DNS entries on network http://blog.justinlintz.com/2010/05/check-for-missing-reverse-dns-entries-on-network/ http://blog.justinlintz.com/2010/05/check-for-missing-reverse-dns-entries-on-network/#comments Wed, 26 May 2010 18:15:48 +0000 justin http://blog.justinlintz.com/?p=90 A quick way to check your network for IPs missing reverse DNS entries….

    nmap -PE -sP 10.0.0.0/24 | awk '{if ($2 ~ /^[[:digit:]]/ ) print $2}' ]]>     A quick way to check your network for IPs missing reverse DNS entries….

    nmap -PE -sP 10.0.0.0/24 | awk '{if ($2 ~ /^[[:digit:]]/ ) print $2}'
    ]]>     http://blog.justinlintz.com/2010/05/check-for-missing-reverse-dns-entries-on-network/feed/ 4     Support for authorized-keys.d/ http://blog.justinlintz.com/2010/02/support-for-authorized-keys-d/ http://blog.justinlintz.com/2010/02/support-for-authorized-keys-d/#comments Mon, 15 Feb 2010 05:02:41 +0000 justin http://blog.justinlintz.com/2010/02/support-for-authorized-keys-d/ Why is there no subdir inside .ssh called authorized-keys.d where I can just throw my ssh keys and easily manage them by file name instead of having to edit the authorized-keys(2) file?

    I need to do some googling on this , a quick search yields this debian bug report on wishing for support for . . . → Read More: Support for authorized-keys.d/]]> Why is there no subdir inside .ssh called authorized-keys.d where I can just throw my ssh keys and easily manage them by file name instead of having to edit the authorized-keys(2) file?

    I need to do some googling on this , a quick search yields this debian bug report on wishing for support for one.

    ]]> http://blog.justinlintz.com/2010/02/support-for-authorized-keys-d/feed/ 0     Windows 7 64bit + iPhone sync http://blog.justinlintz.com/2010/01/windows-7-64bit-iphone-sync/ http://blog.justinlintz.com/2010/01/windows-7-64bit-iphone-sync/#comments Tue, 12 Jan 2010 01:24:55 +0000 justin http://blog.justinlintz.com/?p=78 I somehow attract issues whenever I build a new computer or do a fresh installation on my desktop.  I currently dual boot between Windows XP 64bit and  Ubuntu 9.10 (Karmic) which has served me well the last 3 years.  My current setup is running atop a fakeraid mirror which at the time, was a . . . → Read More: Windows 7 64bit + iPhone sync]]> I somehow attract issues whenever I build a new computer or do a fresh installation on my desktop.  I currently dual boot between Windows XP 64bit and  Ubuntu 9.10 (Karmic) which has served me well the last 3 years.  My current setup is running atop a fakeraid mirror which at the time, was a PITA to get configured as it involved using a live cd and following a really long howto while crossing my fingers it would work.  Today the installer supports fakeraid out of the box so no poor soul would have to go through those installation steps.

    I got a new 1TB hard drive for the holidays and I had been meaning to redo my setup for some time and finally took the plunge this weekend.  I also decided I was going to give Windows 7 a go.  With my current(now old) setup, I couldn’t install iTunes in Windows XP 64bit so I had to do all my iPhone syncing using a 32bit Windows XP install inside a Virtualbox VM inside of Ubuntu.  I would run into issues anytime I would have to do a firmware upgrade on my iPhone since the iPhone would disconnect from the VM during the firmware upgrade process and the USB filters I had in place to pass through the iPhone to the VM would run into a race condition with the OS while it tried to mount the iPhone as a photo device.  This would end up leaving my iPhone as a brick (but upgraded) and I’d have to restore from backup and re-sync all my music, a process that would take up to 4 hours.  The last time I did an iPhone upgrade, I learned my lesson and did the upgrade from a Mac, no restores needed and I didn’t need to do any syncing after.

    With my new setup I was planning on doing all of my syncing in Windows 7, without having to worry about any issues with USB pass through or performance issues of doing the sync through a VM.  Once I got my Windows 7 install up and running, I installed iTunes and added my music to the library.  I went to go ahead and perform my first sync and noticed it began syncing then about 5 seconds later it would just finish.  I tried this a couple more times and noticed it was still not behaving right.  I figured I’d try to atleast backup my iPhone incase anything went wrong.  I began the backup process and went to go play some CoD MW2.  I came back an hour later and noticed no real progress on the backup of the iPhone.  I cancelled the backup process and then tried restarting the backup.  I let it sit for a couple of hours and it never completed.  Normally a backup will take maybe 10-15 minutes so I knew something was wrong.

    I began Googling and right away I came upon this thread on the Apple forums.  The thread described a varying amount of issues with Windows 7 64bit and iTunes syncing.  Issues with people having their iPhone not recognized to certain chipsets causing sync issues.  I tried jumping around the thread a bit looking for some simple answers related to my issues, but most of the solutions were geared towards BIOS updates.  After some more Googling I found a thread on one of the MS support forums (sorry I lost the link) where a representative from MS was actually contributing to the thread and seemed to be working with developers on a hot fix for the issues.  The thread talked about users with issues with USB hard drives and iPhone issues as well. Later in the thread MS eventually came up with a hot fix for the issue so I figured I’d give it a go.  The hotfix can be found here http://support.microsoft.com/kb/976972 .  I installed it and low and behold, it actually fixed my issue.  This is the first time I can remember downloading a hotfix for an issue I’ve had with Windows and it actually fixing the problem.  I’m now finishing syncing up my music to my iPhone which is syncing A LOT faster than going through a VM layer.  Hopefully someone will stumble across this blog post facing similar issues and it will solve their iTune woes.

    ]]>     http://blog.justinlintz.com/2010/01/windows-7-64bit-iphone-sync/feed/ 2     Reserved block space http://blog.justinlintz.com/2009/10/reserved-block-space/ http://blog.justinlintz.com/2009/10/reserved-block-space/#comments Sun, 04 Oct 2009 23:17:16 +0000 justin http://blog.justinlintz.com/?p=52 Normally when a partition fills up, you remove some files, freeing up some space and watch your disk usage drop in

    df -h

    The first time recently I ran into a situation where that wasn’t the case. I had filled up my root partition and after removing some files and moving some others to . . . → Read More: Reserved block space]]> Normally when a partition fills up, you remove some files, freeing up some space and watch your disk usage drop in

    df -h

    The first time recently I ran into a situation where that wasn’t the case. I had filled up my root partition and after removing some files and moving some others to another partition I noticed no change in my output of

    df -h

    Applications were crashing since they couldn’t write to the partition but I knew I had removed at least a few GB’s worth of data. What gives?

    After some frantic googling I stumbled upon something I hadn’t heard of before, known as “Reserved blocks”. Paraphrasing the man page on tune2fs, Reserved blocks are a set of filesystem blocks set aside for use by privileged processes to avoid fragmentation as well as allow system daemons to function after all available disk space has been used up. By default the filesystem sets aside 5% of the filesystem during formatting for use in reserved blocks. If you have a large hard drive, 5% may be a bit much. You can adjust this amount with the command

    tune2fs -mX /dev/sda1

    where X is the percentage you wish to change to and /dev/sda1 is the partition you wish to modify. In order to get out of the hole I dug myself, I used the command

    tune2fs -r 0 /dev/sda1

    to reset the block count to 0 and thus giving me back my space I had freed up. Be sure that you have freed up space before running this command, as the reason reserved blocks exist is so in the instance when you have completely filled up your partition, you will still be able to run important daemons and commands without them crashing.

    Maybe I will finally add some monitoring to my own desktop to prevent this from happening to me. So far I’ve been bitten by this twice, I noticed each time when pidgin began crashing and firefox wasn’t saving any cookies when logging into any sites.

    ]]> http://blog.justinlintz.com/2009/10/reserved-block-space/feed/ 0     Online backup solution for Linux http://blog.justinlintz.com/2009/10/online-backup-solution-for-linux/ http://blog.justinlintz.com/2009/10/online-backup-solution-for-linux/#comments Thu, 01 Oct 2009 04:42:07 +0000 justin http://blog.justinlintz.com/?p=48 The past couple of weeks I’ve begun looking into an online backup solution as I start to realize I’d be really really really pissed if something happened to my data.  Currently I’m running my desktop in a RAID 1 configuration using dmraid aka fakeraid, under Ubuntu Jaunty 9.04.  The reason I ended up having . . . → Read More: Online backup solution for Linux]]> The past couple of weeks I’ve begun looking into an online backup solution as I start to realize I’d be really really really pissed if something happened to my data.  Currently I’m running my desktop in a RAID 1 configuration using dmraid aka fakeraid, under Ubuntu Jaunty 9.04.  The reason I ended up having to use fakeraid was because I still dualboot Windows XP 64bit for doing photoshop work and the occasional game play.  I could do my photoshop work in a VM, but then I lose the advantage of a dual screen workflow when going through my photos.

    My initial reaction into doing online backups was to use Mozy since they are owned by EMC and I wouldn’t have to entertain the thought of the company going under or running into any growth issues.  They also have the great price point of 4.99/month for unlimited backup storage.  Of course Mozy falls short on the basis that they don’t have a native linux client.  I’d have to run the client in my Windows VM under virtualbox and expose the whole Dom0 filesystem to the VM.  I only fire up my Windows VM as needed and didn’t care to have it running all the time, or forgetting to launch it and missing an important backup.  A couple of other downsides to Mozy,

    • No ability to easily share files
    • Their external HD support is not straight forward, the device must be “fixed” and not “removable media”
    • You have to pay an additional fee for additional computers you want to backup

    Basically for a basic home user, their service is fine but I’m far from that.  Next up were a couple services that actually had native Linux clients, SpiderOak and MemoPal.

    MemoPal unfortunately looks to have abandoned their linux client, which was in beta anyway.  The download page page judging from the packages for Ubuntu 8.04, appears to have not been updated in a year.  Their price of 50 a year for 150GB of storage is fine for me as I won’t be backing up that much data , so I don’t truly need unlimited but it’s always a nice to have.  You can install the client on up to 10 computers to backup their data as well which is fine for me and I’d imagine most geeks.   Ultimately I didn’t end up going with them since their linux client development seems to have halted and trusting my backups to a beta client seems wrong (although i trusted gmail’s “beta” service for my primary email for years).

    Finally that brings me to the one I ultimately ended up choosing, SpiderOak.  A few things that sold me on SpiderOak were

    • A coupon for 30% off , seriously I may not have went with them if I had to pay 10/month for 100GB vs 7/month
    • Linux Client is actively developed, appears to be part of one code base
    • They open source various utilities and API’s for Python https://spideroak.com/code
    • I can have multiple computers
    • It can run headless!  This may prove very useful as I can just run this on my linode virtual instead of backing up to my home machine first which has an upstream capped at 1 MBit/s.
    • One other odd thing that sold me a bit on them was they recently had a storage cluster fail but they didn’t lose any of their customers data.

    Now thanks to Time Warner Cable’s crappy upstream it should only take me a week or so to upload my data initially , around 50GB.  I also signed up for the month by month payments just in case at anytime I decide I want to switch elsewhere, although I hope that isn’t the case as re-uploading all this data isn’t the fastest process.  I know there are a couple other backup options out there that I didn’t mention such as rsync.net , ibackup.com and dropbox, but those either didn’t fall into the definition of a backup solution (mainly sync) or the price point was just way off.

    ]]>     http://blog.justinlintz.com/2009/10/online-backup-solution-for-linux/feed/ 1