Code, mostly

Compiling QR reader for python / Ubuntu 11.10

2011-11-30T14:12:00.000-08:00

Steps that apparently worked to get a QR code reader compiled for python (2.7) un Ubuntu 11.10 64 bit as of 2011-11-30.

(an alternative to the steps here might be this: http://pyqrcode.sourceforge.net/ but that didn't work easily for me in recent Ubuntus)

These instructions:
http://hi.baidu.com/paulau/blog/item/915e860ffbf7032c6059f34c.html

cover most of the steps, but lots of details need changing now vs. 2008.

The PyQrcodec_Linux.tar.gz mentioned in various places on the web has disappeared from www.pedemonte.eu - in fact that whole domain is gone. You can get it from http://gentoo.mirrors.pair.com/distfiles/PyQrcodec_Linux.tar.gz

The instructions above say:

sudo apt-get install g++
sudo apt-get install python-dev
sudo apt-get install libcv-dev libcvaux-dev

in addition you now need

sudo apt-get install libhighgui-dev

Untar and cd into the PyQrCodec folder.

then edit setup.py and add

extra_compile_args = ['-fpermissive'],

before each of the "sources = ..." lines.

then in the shell run (once and once only)

for file in $(grep -lre BEGIN ./*|grep -v svn); do sed -e "s/_BEGIN_/_CV_BEGIN_/;s/_END_/_CV_END_/" $file -i; done

now python setup.py build and sudo python setup.py install
should work (with all kinds of warnings).

The how to use example should be:

import PyQrcodec
size, image = PyQrcodec.encode('www.example.com')
image.save('example.png')
status, text = PyQrcodec.decode('example.png')
print(text)

.encode() params are:

Definition:     PyQrcodec.encode(string, image_width=400,
  case_sensitive=True, version=5,
  error_correction_level='QR_ECLEVEL_L',
  encoding_hint='QR_MODE_AN')
Docstring:
  Returns a PIL image of the QR code generated
  from the given string

Note: for http://upload.wikimedia.org/wikipedia/commons/thumb/9/9b/Wikipedia_mobile_en.svg/220px-Wikipedia_mobile_en.svg.png I had to flatten the image (replace the transparent background) for decode() to work.

Can I delete that branch? Check bzr branch relationships.

2011-06-21T07:40:00.000-07:00

If you use bazaar (bzr) and end up with several branches for the same project, you can end up wondering if one branch contains all the commits in another, e.g. you need to check that all the work done in a successful experimental branch has been moved into the trunk.

This small python program does that:

leo.repo> bzrin free_layout trunk
Checking for commits of revs in 'free_layout' in 'trunk'
Status of 'free_layout':
unknown:
  .thumbnails/
  demo.jpg
  nohup.out
Status of 'trunk':
unknown:
  *.g1.dml
Counting revs in free_layout
6026 revs in free_layout
Counting revs in trunk
6683 revs in trunk
All revs in free_layout exist in trunk : OK

`bzrin` checks that all the commits in the "free_layout" branch have been merged into the trunk - in this case they have, and you can safely delete "free_layout".

The code uses `subprocess` rather than the python bzr bindings to do its work, but it gets the job done and has proved very useful for tidying up a directory full of branches for various subprojects.

#!/usr/bin/python
"""Check that the latest commit in bzr branch A exists in bzr branch B
"""

# bzrin2
# Author: Terry Brown
# Created: Mon Sep  8 12:18:21 CDT 2008

import subprocess, sys, os
import tempfile  # ? because subprocess.PIPE hangs in .wait() ?

def emit(s):
    sys.stdout.write(s)

def main():
    branch = tuple(sys.argv[1:3])
    emit("Checking for commits of revs in '%s' in '%s'\n" % branch)

    # show status
    for i in branch:
        emit("Status of '%s':\n" % i)
        cmd = subprocess.Popen(('bzr status '+i).split())
        cmd.wait()

    revs = []

    for i in branch:
        emit("Counting revs in %s\n" % i)
        revs.append(set())
        tmpFile, tmpName = tempfile.mkstemp()
        cmd = subprocess.Popen(('bzr log --show-ids --levels=0 '+i).split(),
            stdout = tmpFile)
        os.close(tmpFile)
        cmd.wait()
        source = file(tmpName)
        for line in source:
            content = line.strip()
            if content.startswith('revision-id:'):
                id_ = content.split(None,1)[1]
                while not line.strip() == 'message:':
                    line = source.next()
                line = source.next()
                msg = []
                while not line.strip().startswith('-'*10):
                    msg.append(line.strip())
                    try:
                        line = source.next()
                    except StopIteration:  # end of file
                        break
                revs[-1].add((id_, tuple(msg)))
        os.remove(tmpName)
        emit("%d revs in %s\n" % (len(revs[-1]), i))

    diff = revs[0].difference(revs[1])

    if not diff:
        emit ("All revs in %s exist in %s : OK\n" % branch)
    else:
        emit ("WARNING: %s contains revs NOT in %s\n" % branch)
        for i in diff:
            emit("%s\n%s\n" % (i[0], ''.join(['  '+m for m in i[1]])))
        emit ("WARNING: %s contains revs NOT in %s\n" % branch)

if __name__ == '__main__':
    main()

Use multiple cores for shell scripts in Ubuntu

2010-10-28T14:04:00.000-07:00

So you want to use all your CPU's cores for some shell based batch processing task. Seems there should already be an app for that, and there is, parallel, in the more-utils package in Ubuntu. But it's not that easy to use the target file argument in a shell script. (note: I think there may be more than one version of this utility, I'm referring to the one that ships with Ubuntu).

For example, I wanted to use all cores for this operation:

for i in svg/*.svg; do f=$(basename $i); \
  inkscape-devel --without-gui \
  --export-background=white \
  --export-ps ps/${f%%svg}ps $i; \
  echo $f; done

So parallel has an -i flag which enables replacement of {} with the target argument ($i in the above) but only if it's surrounded by spaces and not quoted, hardly convenient for scripting. This simple wrapper (saved in a file called task, made executable and placed somewhere on your $PATH) gets around that problem:

# helper for parallel
#
# usage: task 'shell-pattern' 'shell commands'

GLOB="$1"
shift
SCRIPT=$(mktemp)
echo "$@" >"$SCRIPT"
chmod +x "$SCRIPT"
parallel "$SCRIPT" -- $GLOB
rm "$SCRIPT"

So now you can use $1 (not $i) in your shell code without any complications. The above example becomes:

task 'svg/*.svg' 'f=$(basename $1); inkscape-devel \
--without-gui --export-background=white --export-ps \
ps/${f%%svg}ps $1; echo $f'

...and running on four cores it's much quicker :-)

Loading SQL tables column by column

2010-05-26T09:21:00.000-07:00

Goal: Load data copied from an PDF table into a RDMS table column by column, using SQL.

Selecting and copy/pasting the whole PDF table at once didn't extract the data in clean or usable way, things got jumbled. But selecting one column at a time (using xpdf) cleanly extracted the data in that column. But how can you insert it into the table without messing up the ordering of each columns content? OMG! The Excel "reordering destroys data integrity" problem has come to SQL! :-) Anyway, given a table like this:

21	Ant	One
31	Bat	Two
76	Cat	Three
89	Dog	Four

The following approach will work (from a postgres / psql session):

create table rescued_data (
  col1 int,
  col2 text,
  col3 text,
  ordering int
);

create temp sequence s;
create temp table col (val text);

\copy col from stdin
21
31
76
89
\.

insert into rescued_data (col1, ordering)
  select val::int, nextval('s') from col;

-- note need to match type with ::int in the above

select setval('s', 1, false);  -- reset the sequence
truncate col;

\copy col from stdin
Ant
Bat
Cat
Dog
\.

update rescued_data set col2 = val
  from (select val, nextval('s') as seq from col) as x
  where seq = ordering;

-- repeating above for next column

select setval('s', 1, false);  -- reset the sequence
truncate col;

\copy col from stdin
One
Two
Three
Four
\.

update rescued_data set col3 = val
  from (select val, nextval('s') as seq from col) as x
  where seq = ordering;

select * from rescued_data;

-- if necessary, you can
alter table rescued_data drop column ordering;

Python/PyQt upgrade triggers strange bug

2010-05-16T11:56:00.000-07:00

Percy: Look, look, I just can't take the pressure of all these omens anymore!

Edmund: Percy...

Percy: No, no, really, I'm serious! Only this morning in the courtyard I saw a horse with two heads and two bodies!

Edmund: Two horses standing next to each other?

Percy: Yes, I suppose it could have been.

Blackadder, "Witchsmeller Pursuivant"

Today I saw a bug with one head and two bodies. Upgrading from Ubuntu to 9.10 to 10.4 broke a tool bar button in Leo, the world's best code editor / project manager / note sorter. The upgrade involved transitions from Python 2.6.4 -> 2.6.5 and PyQt 4.6 -> 4.7.2. The forward and back browsing buttons supplied by Leo's nav_qt plugin stopped working.

After Brain had been debugging, testing, googling, comparing etc. for over two hours, Intuition wanders past and says, "oh, ha, why not try

def __init__ (self,c):
         self.c = c
+        c._prev_next = self
         self.makeButtons()

Sometimes, Brain doesn't like Intuition very much.

Fortunately Brain was able to save some face, as

-        act_l = QtGui.QAction(icon_l, 'prev', ib_w)           
-        act_r = QtGui.QAction(icon_r, 'next', ib_w)           
+        act_l = QtGui.QAction(icon_l, 'prev', ib_w, triggered=self.clickPrev)   
+        act_r = QtGui.QAction(icon_r, 'next', ib_w, triggered=self.clickNext)

was also required.

So it seems like the upgrade caused two changes which both had the same symptom, making debugging a challenge. It seems like the plugin class instance or the actions it was creating are now being garbage collected where they weren't before. The c._prev_next = self would prevent the instance being collected, although it's unclear that it should also prevent the actions being collected. You would think the GUI's links to the actions would be enough to protect them, so perhaps that bug body wasn't an old glitch going away, but a new one being introduced. OTOH the gui must have a link to the actions, as it's able to trigger them.

The triggered=self.clickPrev addition presumably covers a change in the emission of 'clicked()' by QToolButton, or a change in default actions, or something. Passing the parameter that way is a PyQt alternative to act_r.connect(act_r, QtCore.SIGNAL("triggered()"),self.clickNext), which would probably also have worked.

zipas - ensure .zip file contains an outer wrapper folder

2010-05-12T09:42:00.000-07:00

A trivial bash script to create a .zip file where everything's in
a top-level folder. Saves creating a folder temporarily, and copying
/ linking things into it:

#!/bin/sh

if [ @# -lt 2 ]; then cat <<EOF

usage: $0 <wrapper-folder> <target files>

e.g. zipas jul0907 *.c
EOF
else

  PATHNAME="$1"

  HEADNAME=${PATHNAME%%/*}  # i.e 'foo' from 'foo/bar/inner'

  if [ -a "$HEADNAME" ]; then

    echo "ERROR: '$HEADNAME' exists"
  else
    shift
    mkdir -p $PATHNAME

    for i in $@; do
      ln -s "$PWD/$i" "$PATHNAME/$i"
    done
    zip -r $HEADNAME.zip $HEADNAME

    rm -rf "$HEADNAME"
  fi
fi

posted at: 15:01 |
path: /code/bash |
permanent link to this entry

Merging PDF files

2010-05-11T14:26:00.000-07:00

The python code at the bottom of this posting can be used to merge PDF files (via GhostScript (gs)). In theory gs can do that by itself, in practice I found merging about 160 single page files into one resulted in strange characters appearing in some of the text. The python code merges files two at a time, repeatedly, until all are merged. Merging file 1 and 2, then that with 3, then that with 4, etc. may also work, but it becomes very slow for a large set of files. The binary approach here is much faster.

This code is just a quick hack. If you have a large pile of PDFs to merge and GhostScript is failing as described above, this could save your day. It's invoked from the command line by:

python pdfmerge.py *.pdf

and merges the files in the order listed, creating a lot of files called __XXXX.pdf in the process. The last __XXXX.pdf file produced is your output, you should rename that one and delete the rest. I did say it was just a quick hack :-)

"""Merge pdfs using GhostScript (gs)

Work around for a bug in gs such that::
    
    gs -sDEVICE=pdfwrite -dNOPAUSE -dBATCH -dSAFER 
    -sOutputFile=foo.pdf *.pdf
    
produces odd text corruptions if *.pdf expands to a large number of files.

This program uses a binary merging approach which seems to avoid the bug.
"""
    
import sys
import os
import subprocess
from collections import defaultdict

pages = defaultdict(lambda:1)  # number of pages in each file

pdfs = sys.argv[1:]            # pdfs to merge, already ordered

idx = 0                        # sequence number for temporary pdfs

newpdfs = []                   # list of new pdfs to process

while pdfs or newpdfs:

    if not pdfs:  # pdfs list ends up empty whenever it starts of even length
        pdfs = newpdfs
        newpdfs = []

    if len(pdfs) == 1:
        # only one left, just add it to the end of the list for next iteration
        newpdfs.append(pdfs.pop(0))
        pdfs = newpdfs
        newpdfs = []
        if len(pdfs) == 1:  # we're done
            break

    pdf0 = pdfs.pop(0)  # pair of pdfs to merge
    pdf1 = pdfs.pop(0)

    assert os.path.isfile(pdf0)  # should both exist
    assert os.path.isfile(pdf1)

    idx += 1
    newpdf = "__pdf%04d.pdf" % idx

    pages[newpdf] = pages[pdf0] + pages[pdf1]

    newpdfs.append(newpdf)  # add new pdf to list for next iteration

    cmd = ("gs -sDEVICE=pdfwrite -dNOPAUSE -dBATCH -dSAFER "
    "-sOutputFile=%s %s %s" % (newpdf, pdf0, pdf1))

    # here's a lot of uneeded paranoia that arose when I was feeding in some
    # bad (0 byte) pdf files, doesn't hurt to leave it in

    print pdf0, pdf1, newpdf, pages[pdf0], pages[pdf1], \
        pages[newpdf], len(pdfs), len(newpdfs)
    print cmd

    proc = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE, 
        stderr=subprocess.STDOUT)

    out,dummy = proc.communicate()

    print out
    print

    # the term 'Processing pages ' should occur twice, if both files were read
    procs = [i for i in out.split('\n') if i.startswith('Processing pages ')]

    assert len(procs) == 2

Syntax highlighting for Pyblosxom, S5, etc.

2009-08-15T13:41:00.000-07:00

Oct. 2010 - note this also works for apps. like rst2s5 - just make your own version of rst2s5 which includes the required module as shown below. rst2s5 is a wrapper script of just a few lines, so it's easy to do.

After running around in circles a bit I've found it's quite easy to
get syntax highlighting in Pyblosxom if you're using the rst plugin.

Install the rst plugin.
Make sure pygments is installed on your system. It's available as
a package for Ubuntu, so you can just do:
sudo apt-get install python-pygments
Get rst-directive.py from the pygments distribution (or from the
end of this article). It's
in the 'external' folder. It may not be included in the package
for your system, but you can get the file by itself by browsing
the pygments site.

Rename it rst_directive.py because that seems more sensible,
and put it in you plugins folder. You would think you would need
to import it from the rst plugin, but you don't, because
Pyblosxom will have imported it just because it's in the plugins
folder.
Edit rst_directive.py and change:
INLINESTYLES = False
to
INLINESTYLES = True
Doing so causes pygments to use <span style="color: #BA2121">

instead of <span class="comment"> etc., but it saves a lot of
trouble trying to ensure the CSS file is available.

That's it. Now in your rst you can use the sourcecode directive:

That's it.  *Now* in your rst you can use the `sourcecode` directive:

.. sourcecode: python

  import foo  # get access to foo
  foo.bar("test") or raise SystemExit

to syntax highlight rst (above), python (below), and all the other languages and markups pygments knows.

import foo  # get access to foo

foo.bar("test") or raise SystemExit

Graham Higgins's post may have set me off on the right track here, at least by making me aware of pygments.

Syntax highlighting on external files

To make pygment's sourcecode directive even more useful I added a
quick tweak to allow inclusion of a file like this:

.. sourcecode:: python
  :filename: ../tlog/plugins/rst_directive.py

In this form the inline content is ignored and the content of

filename is formatted instead.

Here's the complete, modified, rst_directive.py (original from external
folder on pygments Trac site). Put it in your
plugins directory. Modifications are:

INLINESTYLES = True

the if 'filename'... block

setting ...options['filename'] = directives.path

# -*- coding: utf-8 -*-
"""
    The Pygments reStructuredText directive
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    This fragment is a Docutils_ 0.4 directive that renders source code
    (to HTML only, currently) via Pygments.

    To use it, adjust the options below and copy the code into a module
    that you import on initialization.  The code then automatically
    registers a ``sourcecode`` directive that you can use instead of
    normal code blocks like this::

        .. sourcecode:: python

            My code goes here.

    If you want to have different code styles, e.g. one with line numbers
    and one without, add formatters with their names in the VARIANTS dict
    below.  You can invoke them instead of the DEFAULT one by using a

    directive option::

        .. sourcecode:: python
            :linenos:

            My code goes here.

    Look at the `directive documentation`_ to get all the gory details.

    .. _Docutils: http://docutils.sf.net/
    .. _directive documentation:
       http://docutils.sourceforge.net/docs/howto/rst-directives.html

    :copyright: Copyright 2006-2009 by the Pygments team, see AUTHORS.
    :license: BSD, see LICENSE for details.

"""

# Options
# ~~~~~~~

# Set to True if you want inline CSS styles instead of classes
INLINESTYLES = True

from pygments.formatters import HtmlFormatter


# The default formatter
DEFAULT = HtmlFormatter(noclasses=INLINESTYLES)

# Add name -> formatter pairs for every variant you want to use
VARIANTS = {
    'linenos': HtmlFormatter(noclasses=INLINESTYLES, linenos=True),
}



from docutils import nodes
from docutils.parsers.rst import directives

from pygments import highlight

from pygments.lexers import get_lexer_by_name, TextLexer

def pygments_directive(name, arguments, options, content, lineno,
                       content_offset, block_text, state, state_machine):

    if 'filename' in options:
        if options['filename']:
            content = [line.rstrip('\n') for line in file(options['filename'])] 
        del options['filename'] 

    try:
        lexer = get_lexer_by_name(arguments[0])
    except ValueError:
        # no lexer found - use the text one instead of an exception

        lexer = TextLexer()

    # take an arbitrary option if more than one is given
    formatter = options and VARIANTS[options.keys()[0]] or DEFAULT
    parsed = highlight(u'\n'.join(content), lexer, formatter)
    return [nodes.raw('', parsed, format='html')]

pygments_directive.arguments = (1, 0, 1)
pygments_directive.content = 1

pygments_directive.options = dict([(key, directives.flag) for key in VARIANTS])

pygments_directive.options['filename'] = directives.path

directives.register_directive('sourcecode', pygments_directive)