Introduction

For camping gear this means you can’t…

• Yank on a zipper to see if it’s sturdy.
• Set up a tent to see if your other gear will fit inside.
• Check tags for “Made in the USA”.
• Try to fit your camp stove inside your cooking pot for storage.

The next best thing to physical inspection are high-resolution product photos. Since many online retailers don’t provide these, I thought I’d share mine. Below you’ll find some pictures I took of the gear I bought last Spring.

I ordered most of my equipment from Campsaver.com, MooseJaw.com, and a used gear broker GearTrade.com and was pleased with the service from each. I spent some time searching Craigslist too. Google Product Search was handy for comparing prices.

Be sure to check RetailMeNot.com for coupons and discount codes. A simple Google search like “campsaver coupon” can be surprisingly effective. Campsaver sent me a “10% off your next order” certificate and the MooseJaw catalog contains lots of codes redeemable for store-brand gear.

Of course, nothing beats advice and hand-me-downs from the pros. Thanks Matt and Kenneth!

I documented the nitty-gritty of my gear requirements at this wiki page. It contains notes specific to planning a trip on the Appalachian Trail too.

I attended last year’s edition with Carolyn just prior to a return trip which included visits to…

• Florida State University in Tallahassee, FL
• Georgia Tech in Atlanta, GA
• UNC Charlotte, NC
• Stony Brook on Long Island, NY

Take a look at the pictures I took at the festival:

These are some photos from the remainder of the trip.

The problem

Have you ever named a file something like…

main.cpp.GOOD

… to indicate a “known-good” copy of `main.cpp`?

This simplistic method for storing file metadata is great because it’s filesystem-agnostic – the metadata (e.g. “this is known-good source code”) “follows” the file wherever it goes – from ext3 to FAT32 to NTFS, etc.

However, it has a few limitations:

1. Certain characters like “/”, ” “, and “?”, while possible, become a hassle later when you need to manipulate the file by name at a shell.
2. Adding, editing, and removing metadata at a shell involves lots of line-editing acrobatics especially if you want to maintain the file’s orginal extension e.g. `main.WORKING.cpp`.

To work around these limitations I wrote a small utility called `fnmeta.py` (for “filename metadata”). It uses Base64 encoding to store metadata, thus allowing you store arbitrary binary data. Beware the limits on filename length imposed by the filesystem. For the filesystems I use, the lowest common denominator is 255 characters.

If you ask `fnmeta.py` to store the string “foo” as metadata for the file `bar.baz`, it will rename `bar.baz` to `bar.TAG.ENCODED_METADATA.TAG.baz` where…

• “TAG” is the word “fnmeta” followed by a number indicating which version of fnmeta was used to encode the metadata. “fnmeta” gives a hint at why there’s a big chunk of non-english text in the filename and the number facilitates future updates to the encoding scheme.
• “ENCODED_METADATA” is base64_encode(“foo”)

Here’s an example showing how I used `fnmeta.py` to store the URL from which I downloaded a PDF named `xapp374.pdf`:

$ fnmeta.py xapp374.pdf
please enter the new metadata: www.xilinx.com/support/documentation/application_notes/xapp374.pdf

$ ls xapp374*
xapp374.fnmeta1.d3d3LnhpbGlueC5jb20vc3VwcG9ydC9kb2N1bWVudGF0aW9uL2FwcGxpY2F0aW9uX25vdGVzL3hhcHAzNzQucGRm.fnmeta1.pdf

$

Later on, I retrieved the metadata like this:

$ fnmeta.py xapp374.fnmeta1.d3d3LnhpbGlueC5jb20vc3VwcG9ydC9kb2N1bWVudGF0aW9uL2FwcGxpY2F0aW9uX25vdGVzL3hhcHAzNzQucGRm.fnmeta1.pdf
www.xilinx.com/support/documentation/application_notes/xapp374.pdf

$

`fnmeta.py` is licensed under the GPL and can be found here.

Anyways, the offer boasted…

• A HASP HL “Time” dongle with built-in AES, host-accessible 4kB EEPROM, and “tamperproof” real-time clock (RTC).
• A cross-platform SDK with libraries and example code in C and a handful of other languages.
• Printed API documentation.

After pestering them with a few emails over the course of a week, AKS sent me the kit and it arrived just in time to distract me from the first round of exams during my Senior year `X|`

The SDK includes a demo application which can…

• Read/write the token’s EEPROM.
• Ask the token to encrypt/decrypt data with its key.
• Read the RTC.

To prevent its use in a commercial setting, AKS hard-codes the encryption keys on its evaluation kit tokens.

So, I set out to make use of one of the ”un-crippled” features: the on-token EEPROM. Claiming 1,000,000+ read/write cycles, it seems like a nice place to store ”my own” encryption keys or perhaps store two-factor authentication data.

I hacked up the demo application to implement a bare-bones FUSE filesystem – one which provides a single file, `hello`, backed by the token’s EEPROM. I call the result “haspfuse“.

Here are its limitations. See the haspfuse code for details.

• The filesystem only supports a single file. This wouldn’t be too bad if you were to stack a fuse-zip or archivemount-based tarball-backed filesystem on top. Don’t even think about JFFS(2) or ext2 – they each require more than 4kB for a single block.
• The single file has a fixed size of 3584.
• No operation besides: `getattr`, `readdir`, `open`, `read`, `write`, `truncate`, and `chmod` is supported.
• The filesystem probably isn’t “eject-safe”. See FUSE’s `direct_io` for a starting point.

Additionally, a proprietary blob “driver” named `aksusbd` and a static library from the SDK named `libhasp_linux.a` are required. I had trouble with version 3.5.0 so I included v1.8.1 with the haspfuse code.

`aksusbd` is a userspace driver that employs usbfs to interact with the token. Unfortunately, the deprecation of usbfs is apparently now complete with the release of the 2.6.31.20 Ubuntu kernel. This means that the old standby methods for re-enabling it no longer work.

You’ll need to use a custom kernel with usbfs support, or better yet, develop an open source driver for these tokens!

The latter approach would be especially helpful because my friend Bob claims that “`aksusbd` is setuid root and full of vulnerabilities”. But that’s another story…

Be aware that various AKS licenses and draconian US laws may prohibit you from reverse engineering `akusbd`.

• Filesystem design for severely space-constrained storage devices.
• How to build a virtual filesystem using the FUSE C API.
• Methods for rebuilding ELF symbol tables and the idiosyncrasies that result when the binary is dynamically-linked and uses the Native POSIX Thread Library (NTPL).
• The software tools available for USB protocol reversing.
• The myriad sorts of implementation failures that lead to “cracked” hardware tokens.

• etokenonlinux.org – A great resource for using other AKS tokens to do two-factor authentication, one-time password storage, and encrypted partition unlocking The Right Way™. Provided by Cornelius Koelbel.
• Guy-Gregoire Leclercq’s detailed notes on using the eToken 64k with OpenCT/OpenSC on Debian.
• Andy Smith’s rebuilt, OpenSC-enabled, openssh-client and how to store SSH keys on an eToken with it.

1. Well, mortals too squeamish to run Plan 9 as their every-day operating system ;]

Well too bad -- it’s just not possible to ascertain details that are hidden from the camera’s view!

You can get pretty close, though, with some software I discovered via an article in the Linux Journal.

GDV uses the Visualization Toolkit (VTK) to display what are called “surface plots”. I wanted to record a simple screencast of myself rotating a surface plot by hand but I couldn’t convince VTK to do “live” previews. As a workaround I whipped up this shell script which rotates the surface plot in small increments, taking screenshots along the way:

# simulate multiple click-and-drag mouse events along predetermined

# coordinates. after each of these click-and-drag events, take a screenshot
# with imagemagick's "import" utility.

# you should generate coordinates at which to click with...
#  (while :; do xdotool getmouselocation | \
#   awk '{print substr($1,3) " " substr($2,3)}'; done) | uniq
# ... and save them to the file at COORDS_PATH

# you can assemble the screenshots into a movie with...
#  ffmpeg -r 30 -b 300k -i %05d.jpg -b 1157kb out.mp4

COORDS_PATH=~/prog/sh/screenshot_rect_while_dragging.dat
SHOTS_DIR=/media/humid_data/tmp/shots
# head -n1 ${COORDS_PATH} | read PREV_X PREV_Y
# PUZZLE: why doesnt the above work? instead we'll use...
PREV_X=343; PREV_Y=135
I=1
mkdir ${SHOTS_DIR}
while read CUR_X CUR_Y
do
  echo moving to ${CUR_X} ${CUR_Y} as step ${I}
  # move to pos
  # greets to fellow RIT alum jordan sissel, xdotool's author
  xdotool mousemove ${PREV_X} ${PREV_Y}
  # mouse down
  xdotool mousedown 1
  # move to pos
  xdotool mousemove ${CUR_X} ${CUR_Y}
  # mouse up
  xdotool mouseup 1
  # give the app some time to finish rendering
  sleep .3s
  # take screenshot
  import -crop 640x480+38+130 -window root -quality 100 \
  ${SHOTS_DIR}/$(printf "%05d" ${I}).jpg

  I=$((I+1))
  PREV_X=${CUR_X}
  PREV_Y=${CUR_Y}
done < ${COORDS_PATH}

Next I assembled the screenshots into a video with Openshot, a young but very stable and featureful nonlinear video editor. Here’s the result:

www.youtube.com/watch?v=a4UCOrIZb2s

For some more image-processing fanciness, see this slick video demo of the “structural editing” tools planned for Photoshop CS5. There’s interesting discussion (incl. some NSFW comments) at this proggit thread. For instructions on feeding “heightmaps” to a 3D printer, see these instructions.

We stopped at the Museum of Science as soon as we arrived. It housed an excellent “history of mathematics” exhibit sponsored by IBM and sprinkled with comic strips to keep it from getting too heavy. I became comfortably lost there for almost an hour.

I stopped by the computing exhibit next and found a replica Enigma Machine -- a piece of WWII crypto history that was central to the plot of a book I just finished -- Neal Stephenson’s Cryptonomicon. The machine itself was understandably enclosed in a plastic display case, but an authentic Enigma transmission bleated out from a little speaker at the push of a button.

Around the corner I found “Whirlwind“, an early computer built down the river at MIT under an Air Force contract. The machine included a shoebox-size stack of “core” memory -- you know, the stuff made of ferrite donuts after which most Unixes name the memory dumps created for misbehaving programs¹. The museum installed a 16-bit bank of interactive core memory nearby that you can fiddle with . Dime-sized compasses indicate the values programmed at each bit. Neat.

www.youtube.com/watch?v=lnvWbq6V7AU

www.youtube.com/watch?v=K89lFWV2KcI

We consulted Yelp for dinner ideas and decided on Cafe Belo, a Brazilian BBQ restaurant in Everett of the rodizio style. You pay a fixed price ($10, there) and servers stop by your table every few minutes with hot meats on skewers, asking if you’d like a slice. Brazilian Beef ribs, Jamaican chicken, sirloin, pork, and some killer garlic beef.

On Saturday I got up early and took the train to Cambridge to visit the MIT campus.

It was quiet save for a scattered handful of undergrads with MIT sweatshirts looking more energetic than any undergrads I’ve seen on a weekend morning.

Access was restricted to Rivest’s hallway, the Program Analysis Group’s wing², and the Computer Systems Security Group’s wing. I did, however, have lunch in the curiously-shaped Stata center and stumble across a Lisp Machine!

That afternoon we drove to the IMAX theater in Natick to see Tim Burton’s Alice in Wonderland. An enterprising furniture salesman convinced IMAX to build this theater in the middle of his showroom, thus guaranteeing a continuous flow of people in a receptive frame of mind. Ingenious.

On Saturday night we met Jumbo’s high school buddy, Mike, at his home a few blocks from Bunker Hill. We all went out to celebrate Mike’s birthday at the Midwest Grill. It turned out to be another Rodizio restaurant. In addition to Cafe Belo’s fare, we had bacon-wrapped chicken, sirloin tips, checken breasts, chicken hearts (!), and some of the best keilbasa I’ve ever had. The hot buffet included some amazing smooth and buttery ocra, two stews, and some mean pork rinds.

As if to prevent me from floating off into culinary heaven, our waitress denied my request for draft Sam Adams (imagine that -- out of Sam Adams in Boston) and instead brought me a bottle.

We returned around noon today after having some killer french toast at Ball Square Cafe in Everett and climbing the Bunker Hill Monument.

1. Linux 2.5+ gives you some options in this regard via its core_pattern `/proc` knob
2. Check out the deep reading list at the Program Analysis Reading Group’s site

1. Software that you run on your computer (the “host”) to send commands to…
2. A USB “token” which A) encrypts and stores credentials from the host and B) decrypts and displays them on its LCD.

Creds101 serves the same purpose as traditional “password database” solutions but it is different in two fundamental ways:

1. Credentials aren’t stored on the user’s computer – they’re stored on a USB “token”.
2. Sensitive credential data ”cannot be read by malicious software on the user’s computer” – credentials can only be viewed on the token’s “trusted LCD display”.

• AsciiDoc-based documentation including the “host->token” serial command protocol and data-flow diagrams illustrating how credentials are encrypted, stored, decrypted, and displayed by the token.
• A unittest test suite which exercises each of the commands accepted by the token. Install Twisted and try

  PYTHONPATH=src trial src/TestSimulatorToken.py

• A cross-platform “token administration” GUI for storing and retrieving credentials. Try

  bin/creds101-admin-gui --use-simulator

  or see screenshot 1.

  
• A command-line “token administration” interface. Try

  bin/creds101 --help

• A “token simulator” that allows us to 1) develop tests 2) iron out token design issues ”before” coding in C for an embedded platform. Try

  bin/creds101-simulator --gui

  or see screenshot 1.

• 
  The beginnings of the code for an Arduino-based token. Take a look in `src/arduino`. This includes code for…
  • Receiving commands from the “host” via the Arduino’s FTDI serial-over-USB chip.
  • Reading and writing data to the 512 bytes of internal EEPROM or an attached SPI DataFlash part. See photos 2 and 3.
  • Displaying text on a SparkFun SerLCD LCD module.
  • Encrypting and decrypting data with 128-bit AES.

• 
  Completing the Arduino token implementation so that we have something that’s suitable for everyday use.
• Completing a thumbdrive-sized token implementation – the ideal form factor.
• The smaller tasks that can be found in the `TODO` file.

• How to create slick, source-based docs with the AsciiDoc tool suite. As much as I like MoinMoin markup, the wiki engine itself isn’t amenable to being invoked from a build automation tool.
• GUI design with Glade, a slick RAD tool used by many free software projects.
• GUI interaction with PyGTK. I wrote a well-behaved worker thread that runs alongside the GTK main loop for the token simulator component.
• The theory and implementation of “Secret Sharing” schemes especially Shamir’s Secret Sharing Scheme (SSSS). We discarded SSSS in favor of the current “store-the-key-on-the-host, store-the-ciphertext-on-the-token” design.
• A handful of Python skills…
  • Finding a script’s location ”from within the script”. Handy for using `bin/foo -> src/foo.py` symlinks in conjunction with external (e.g. Glade XML) resources.
  • Python’s built-in XML-RPC library. Reconciling UNIX socket semantics with thread semantics gave me fits before I switched to this IPC mechanism.
  • Steven Bethard’s superior argparse module which makes handling subcommands and required positional arguments a breeze compared with Python’s standard optparse module.

Creds101 is licensed under the GPL.

Here are the steps to grab the v0.1 source tarball from the hgweb interface and take the GUIs for a test drive!

$ wget http://unsyncopated.com/hg/creds101/archive/0.1.tar.gz
$ tar xzf 0.1.tar.gz
$ cd creds101-0.1
$ bin/creds101-simulator --gui &           # launch the token simulator
$ bin/creds101-admin-gui --use-simulator & # connect to the simulator

You can see our design notes, plans, and a list of similar projects at the wiki page but beware the outdated Secret Sharing content.

I’m accumulating all the research papers, datasheets, and application notes I’ve encountered during this project in a “research materials” repository. You might find the contents useful if you’re trying to choose an AES implementation for an AVR microcontroller or if your project falls into one of these categories: “cryptographic co-processor”, “hardware security module”, “embedded secret secret sharing”, “ubiquitous/pervasive computer security”, or “tamper evidence/proof”.

“A5/1 [the encryption scheme used in most cellular voice calls] has operated unchanged for the last 21 years but it has now reached its cryptographic end-of-life, engulfed by the march of Moore’s Law. However, the operational end-of-life of A5/1 may still be decades away as there are approximately 2 billion GSM subscribers, commanding about 80% of the global mobile market. This would be a tough product recall indeed. A5/1 is well-positioned to become the NT of the mobile crypto world, and I see the makings of a long tail of GSM vulnerability.” - Dr. Luke O’Connor at NoTricks: Another crack at open Rainbow Tables for A5/1

The ability to intercept and decrypt GSM cell phone conversations is now well within the reach of hobbyists.¹²

My friend Scott and I were discussing this sobering fact one night and we began wondering if any systems exist which provide end-to-end encryption for this insecure link.

Sure, there are plenty of solutions for people with access to cellular dataconnections, but what can voice call participants use to foil eavesdroppers?

We didn’t find any low-cost systems so we decided to create our own. OKCrypto is the Linux-based encrypting software modem that we’ve made. It consists of two components: the modem component and crypto component.

We needed the ability to send data before we could try sending  encrypteddata, so the first step was to design a simple software modem.

Rather than executing the modem code on the cell phones themselves, I decided to host the modem code on a the sender and receiver’s Linux systems. This design provides two benefits:

1. The modem code and crypto code has access to the rich Linux API.
2. Sensitive code and data are isolated from the both the cell phone itself and from the cell infrastructure. We have clean separation between trusted (PC) and untrusted (cell) environments.

Thus, the “real work” in OKCrypto is done on the PC – the cell phones simply allow the PCs to talk to each other. See figure 1 for an overview of the hardware involved in the system.

Conceptually, the modem is similar to the analog Plain Old Telephone Service (POTS) modems from the Bad Old Days before broadband service became popular. Instead of an audio coupler, we’re using Bluetooth to connect our PC to our phone line. Instead of custom hardware, our modem is comprised of some glue scripts and the software packages they connect – all running on a Linux PC.

The modem doesn’t provide an asynchronous full-duplex communication link like traditional modems. This modem’s operation is simpler: The sender’s modem dials the number of, and subsequently transmits a pre-prepared chunk of data to, the recipient’s modem. It then hangs up.

The modem uses Dual-tone multi-frequency (DTMF) signaling to encode the data it transmits. I chose DTMF because I was familiar with it and because Debian provides a package for Multimon. OKCrypto uses two utilities that Multimon provides:

1. gen – a DTMF generation utility (digits->wav file)
2. multimon – a DTMF detection utility (wav file->digits)

I’d been working on a few Bluetooth security projects at the time, so the Hands-Free Profile (HFP)³ came immediately to mind as a convenient way to transfer audio (and any data we’ve encoded in the audio) between a PC and a cell phone during a call.

-flickr size=”small” float=”right”-4153181752-/flickr-In most cases HFP is used to connect a a Bluetooth phone to a Bluetooth headset so that the headset can be used to make calls via the phone. See figure 2.

-flickr size=”small” float=”right”-4153181518-/flickr-In the PC world, HFP is typically used to connect a desktop computer to a bluetooth headset. See figure 3. In that configuration, the PC fulfills the first of two roles mandated by the HFP specification: the Audio Gateway (AG) role. The headset fulfills the hands-free (HF) role.

Support for the HF role in bluez, the Linux Bluetooth stack (pronounced “blue-zee”), is just now maturing⁴ so I went searching for a userspace implementation of the HFP protocol stack.

One of the best, chan_mobile, is distributed as an add-on to the popular open-source private branch exchange (PBX) Asterisk system. If you configure chan_mobile to use your cell phone, Asterisk can make both outbound calls and receive inbound calls with the phone.

Asterisk is the largest software component of the OKCrypto modem. It not only provides a reliable HFP HF role implementation which works with a wide array of modern phones (see above), but also many essential telephony operations:

1. Recording audio during a call. OKCrypto uses Asterisk’s built-in voicemail capabilities.
2. Transmitting audio during a call.
3. Pausing for a given time period.
4. Logging phone call details.

See figure 5 for an overview of the software components in the OKCrypto system. Note that the same software is used on both the sender and receiver’s PCs.

Let’s look at how we can use this modem to send a 16-byte binary file over a cellular voice connection.

First, the sender and recipient will need to perform some set-up steps:

1. 1. Acquire a Linux-supported computer and Bluetooth adapter. I found that using a virtual machine introduces latency that bluez/btusb cannot tolerate.
2. Acquire a HFP-capable cell phone.
3. Install Linux, SoX, Python, multimon, Asterisk, the Asterisk “add-ons”, GPG, and the OKCrypto scripts.
4. Pair cell phone with computer. Grant HFP access.
5. Configure chan_mobile
6. Start Asterisk Next, the sender can issue this command at a shell to send the file ‘/tmp/foo’ to the recipient at 585-555-3258.

   $ ./bin_to_int_seq.py /tmp/foo | ./ast_send.sh 5855553258 [...]
   $ cksum /tmp/foo 668417501 16 /tmp/foo

   She’ll notice her PC making a call with her phone, silently transmitting the data, and disconnecting. Behind the scenes, the OKCrypto scripts will…

7. Convert the bytes in /tmp/foo to a series of decimal digits.
8. Encode the digits as DTMF tones with gen from the multimon package. 1. Increase the pitch (time-independent) to prevent any intermediary systems (esp. Asterisk) from interpreting the tones.
9. Queue the final audio file for transmission by Asterisk.

The recipient will hear his phone ring once before his PC answers the call, records the audio, and disconnects. When the call is complete, Asterisk will invoke one of the OKCrypto scripts to…

1. Decrease the pitch to yield the original DTMF tones.
2. Decode the DTMF tones to a series of decimal digits.
3. Convert the digits to a series of bytes which is written to ‘/tmp/bar’.

   $ tr -d '\n' < /tmp/newest_vm.txt | ./int_seq_to_bin.py /tmp/bar [...]
   $ cksum /tmp/bar 668417501 16 /tmp/bar

Now that we can reliably send data, let’s make sure that it’s encrypted first. This turns out to be one of the simplest components of the system – many good crypto APIs are available.

I use GnuPG:

$ gpg --symmetric --force-mdc --cipher-algo AES256 filetoencrypt 

The `–force-mdc` option provides integrity checking – useful for handling transmission errors. Consider these GPG options carefully and make sure they fit your requirements.

Here are a few ideas we’re pursuing for the future of this project:

• Moving the modem code to the cell phone. This would simplify the setup but potentially risk security.
• Improving modem error rate. Data is often erroneously duplicated during transmission.
• Increasing the modem throughput. The current code averages a meager 10 bytes/second.
• Hiding the data within a steganographic channel in a normal voice conversation.
• Incorporating GPG into the OKCrypto scripts.
• Packaging the system as a LiveCD/Live flash drive.
• Implementing key exchange.
• Building an embedded device dedicated to OKCrypto.

I gained experience in the follow areas during the design, implementation. and testing:

• GSM, CDMA crypto. I suspect the cellular phone industry would make a great case study in protocol security by obscurity.
• Cellular voice codecs used by large carriers. Trivia: your calls only require ~10kbps.⁵
• Bluetooth HFP specification and available implementations.
• Debugging latency tolerances in virtual machine USB “passthrough” subsystems.
• The Twilio telephony API. I used Twilio when I only had one cell phone to debug with.
• Asterisk administration.

I’ve described a method to securely transmit data over any of the widely-available cell voice networks. The implementation requires only commodity hardware, open-source software, and minimal setup.

Be aware that transmitting data by “automated means” may violate the terms of your cellular service contract. I disclaim all liability. This information is provided for educational purposes only.

You can download the OKCrypto system here: http://www.unsyncopated.com/corral/okcrypto_v0.1.tar.gz

It is licensed under the LGPL.

• You’ll find lots of links to cell security architecture articles, Bluetooth HFP implementations, and Asterisk administration web pages on the wiki at Crypto Phone/Stacked Linux-based CPhone Brainstorming

Update 1/13/09 - Check out some technical notes on our recent progress with a faster and more reliable modem: OKCrypto/Progress Report for 01-09-2010

1. Open-Source Effort to Hack GSM John Blau – IEEE Spectrum Magazine – December 2009 issue
2. Subverting the security base of GSM Karsten Nohl – Hacking at Random – 8/15/2009
3. Bluetooth Hands-Free Profile (HFP) 1.5 – Bluetooth Special Interest Group – 11/25/2005
4. Comment #1 on Maemo bug #2754 – Johan Hedberg – 1/25/2009
5. Adaptive Multi-Rate audio codec – Wikipedia – 11/24/2009

The problem

You want to create local mirrors of the apt repositories that you use but you don’t have enough hard drive space to mirror every package. Or maybe you have a slow link and you don’t want to spend time downloading packages that you’re unlikely to need.

Only mirror packages whose popularity (as reported by popcon’s “installed” metric) matches a certain threshold.

I’ve been hacking without a network connection recently and one of the biggest pain points is not having access to my distro’s software package repository.

For example, while writing some Python screen-scraping code last week I realized I didn’t have the Python library I wanted to parse some HTML with – Beautiful Soup. Rather than postpone my work on the script until I found a weefee signal, it would have been nice to simply install the package from a local mirror of the repository.

I soon discovered two common tools that can be used to create a local mirror of a repository – Frans Pop’s debmirror and Dmitriy Khramtsov’s apt-mirror.

I chose apt-mirror, skimmed Alan Pope’s handy step-by-step guide and kicked off the mirror script…

$ sudo -u apt-mirror apt-mirror

[...]
52.7 GiB will be downloaded into archive.
Downloading 75 archive files using 10 threads...

ACK! That’s a lot of gibibytes.

Eventually I’d like a complete mirror, but for now, I only want the packages I’m likely to need. My broadband connection isn’t as “broad” as I would like.

The Debian Popularity Contest (“popcon”) came to mind and sure enough, Ubuntu also provides a flat text file containing the names of all packages sorted by the frequency with which they’re installed by users.

I downloaded this file and hacked up the primary apt-mirror perl script to consult the file, only mirroring binary and source packages if they meet a chosen popularity threshold.

Here’s the meat from a patch that applies cleanly to apt-mirror version 0.4.5-1ubuntu2:

sub should_process {
# print "should_process()\n";
my $pkg_name = shift;
my $section_name = shift;
my @popular_pkgs = @{ $_[0] };

# if the pkg isn't in the 'game' section...
if($section_name !~ /game/){
my %is_popular;
for (@popular_pkgs) { $is_popular{$_} = 1 };

if( $is_popular{$pkg_name} ) {
# print "processing popular pkg: " . $pkg_name . "\n";
return 1;
} else {
# print "skipping unpopular pkg: " . $pkg_name . "\n";
return 0;
}
} else {
# print "skipping game pkg: " . $pkg_name . "\n";
return 0;
}
}

# [...]

# open our popcon database
my $db_path = "/home/tz/Desktop/by_inst";
open(FILE,$db_path) or die "Can't open popcon db: $!";
my @data=; # beware record separator ($/) tweak below
close FILE;
my $num_comment_lines = 11;
my $threshold = 3000;
my $cur_line;
my @popular_pkgs;
# for each of the first $threshold lines, grab pkg name
foreach $cur_line (@data[$num_comment_lines .. ($num_comment_lines + $threshold)]) {
# print "cur_line: $cur_line";
my @tokens = split / +/, $cur_line;
# print "pkgname: " . $tokens[1] . "\n";
push( @popular_pkgs, $tokens[1] );
}

# [...]

if( should_process( $lines{"Package:"}, $lines{"Section:"},@popular_pkgs ) ) {
add_url_to_download($uri . "/" . $lines{"Directory:"} . "/" . $file[2], $file[1]);
}

Tweak the path to the flat file (`$db_path`) and the threshold (`$threshold`!) to suit your needs.

 ________________________________________
/ As you can see, I also modified the    \
| script to skip games. Games tend to be |
| large and there aren't many that I use |
\ often, except perhaps cowsay(1) :]     /
 ----------------------------------------
    \               ,-----._
  .  \         .  ,'        `-.__,------._
 //   \      __\\'                        `-.
((    _____-'___))                           |
 `:='/     (alf_/                            |
 `.=|      |='                               |
    |)   O |                                  \
    |      |                               /\  \
    |     /                          .    /  \  \
    |    .-..__            ___   .--' \  |\   \  |
   |o o  |     ``--.___.  /   `-'      \  \\   \ |
    `--''        '  .' / /             |  | |   | \
                 |  | / /              |  | |   mmm
                 |  ||  |              | /| |
                 ( .' \ \              || | |
                 | |   \ \            // / /
                 | |    \ \          || |_|
                /  |    |_/         /_|
               /__/

• Download popcon db file, rather than expect that it already exists on disk.
• Read desired popularity threshold from mirror.list rather than using a hard-coded value.
• Read desired sections as above.
• Speed holes! My perl-fu is weak.

• I found a faster mirror half-way through creating my local mirror. Renaming `/var/spool/apt-mirror/{mirror,skel}/${OLD_MIRROR` to `/var/spool/apt-mirror/{mirror,skel}/${NEW_MIRROR}` was sufficient.
• If you try to install a package from your local mirror which doesn’t exist, you’ll get a 404 error – nothing catastrophic happens.
• Beware permissions issues. Avoid running apt-mirror as root rather than the prescribed `apt-mirror` user.
• debmirror has a `–exclude-deb-section` option

1. Find all the photos that you’ve been tagged in and…
2. Download each photo to a chosen directory.

It worked fine for me after I fixed a trivial tkinter bug.

Right now I’m adapting photograbber to download entire albums for me. Here’s the crucial code that requests a collection of “photo” records using Facebook Query Language (FQL):

photos = self.facebook.fql.query("SELECT pid, aid, src_big FROM " \
"photo WHERE pid IN (SELECT pid FROM photo_tag WHERE subject=" + \
str(self.facebook.uid) + ")")

Changing the query to…

"SELECT pid, aid, src_big FROM photo WHERE aid IN (SELECT aid " \
"FROM album WHERE owner IN (SELECT uid FROM user WHERE name=\"" + \
FriendName + "\") AND name=\"" + AlbumName + "\")"

… did the trick. Caveat coder: the album IDs that you see in your browser while surfing Facebook aren’t the same IDs that you should use in your FQL queries. If you slip up, you might receive this misleading message: “FacebookError: Error 600: An unknown error occurred in FQLPhotoTable::get_ids_for_queries: should never have a pid or aid without a uid”

Keep Facebook’s terms of service in mind when interacting with their site – you don’t want a “cease and desist” letter like Vincent Cheung received for his “FaceDown” application.

If all else fails, you might have some luck with a PHP script called “FBCMD” as illustrated here.

I’ll spare you my internet balkanization rant – Fred Vogelstein made the argument far better in the June issue of Wired: Great Wall of Facebook – The Social Network’s Plan to Dominate the Internet – and Keep Google Out.