Invitation: Gnome/Gtk+/GObject Study and Hack Group

I don't know whether desktop programming is dead, but I do know that there are apps that I want that the web can't give me yet.  These are mostly software development-related, but that's what I use my computer for most.

Since I use Gnome on Linux pretty exclusively these days (and I'd like to continue), I'd like to start a study/hack group for Gnome/Gtk+/GObject programming.

I've been watching the GObject platform change quite a bit over the last few years.  Most significant, in my opinion, has been the introduction of GObject introspection (GI), which essentially made it possible to use any C+GObject library in any language, as long as there's a GI binding for that language (and there are for Java, JavaScript, .Net (Mono), Python, Ruby, and more).

My goals are (1) to be comfortable enough with GI programming to quickly create applications that I want, without the GI layer being the roadblock, (2) to be comfortable enough with C+GObject or Vala to translate potentially reusable pieces of GI applications into actually reusable components.  And (3ish) if I can contribute to the examples, tutorials, and docs all the while, that's a win too.

So, if anyone is interested is interested in learning or practicing Gnome/Gtk+/GObject development in a group, hit me up @mjumbewu, or in the comments.  Standing offer, so even if this post is several months old, hit me up.

Thanks!
- Mjumbe

Playing around with mapping & reducing

So, this is a purely technical one, and is mostly for me.  Apologies, in advance.
I've been doing a few experiments with mapping/reducing in Python using the multiprocessing module.

Some things I've learned

Some of these I suspected, but some were surprises to me.

• Passing data between the parent and child processes is slow -- try not to pass big data.
• Splitting across multiple processes actually does speed up computation, as long as you don't have to pass back a lot of info about each thing.
  For example, this could be a good candidate for mapping to processes:
  

  def square_all(nums):
      return sum([i*i for i in nums])
  

  This, maybe not so much:
  

  def square_all(nums):
      return [i*i for i in nums]
  

  The first just returns a single number for each subsection of data. The latter returns a potentially large list.
  
• Using processes and queues directly is more flexible, but mapping over pools saves a lot of typing. For example, these functions accomplish similar things. The first is a function that just runs the processing on a single entire data set. The next two demonstrate running the data in multiple processes. In multi, it splits the data roughly evenly over n processes and immediately runs those processes in parallel. In pooled, it splits the data into n parts and doles each part out over p processes. When n == p, multi and pooled do basically the same thing.

  Note that, in order to accommodate the Queue pattern in multi, I had
  to modify the function being called:

  def square_all(nums, q=None):
      if q:
          q.put(sum([i*i for i in nums]))
      else:
          return sum([i*i for i in nums])
  

• You cannot pass around simple generators (i.e., like (i*i for i in nums)), because they cannot be pickled. There are ways around this.
• Performance of the process pools was more erratic than the manually handled processes and queues. This may be due to memory constraints; for some reason, when my script got down to looping over the pools, the memory usage went way up. I took a screenshot of my system monitor running on half as much data.
• My computer performed about equally well above 3 processes when they were handled manually. I expected that 8 processes would not have worked out well, but I was wrong.

The script

Here's the source of the script that I used to run the experiment:

#!/usr/bin/env python3
#-*- coding:utf-8 -*-

from itertools import chain
from math import floor
from multiprocessing import Pool, Process, Queue

# ---------------------------------------------------------------------------
# The data to process.  Why a global?  No reason really.

full_count = 1000000
full_range = range(full_count)

# ---------------------------------------------------------------------------
# The processing function.  Something simple.

def square_all(nums, q=None):
    """
    Simple function to loop over and square all the numbers in ``nums``.
    """
    if q:
        q.put(sum([i*i for i in nums]))
    else:
        return sum([i*i for i in nums])

# ---------------------------------------------------------------------------
# The processing wrappers.

def single():
    """
    Run the the function on the full range of values.
    """
    result = square_all(full_range)
    return result

def multi(n):
    """
    Partition the range into n parts and run each in a separate process.
    """
    # Partition
    parts = _partition(full_range, n)

    # Map
    queues = [Queue() for _ in range(n)]
    processes = [Process(target=square_all, args=(part, queue))
                 for part, queue in zip(parts, queues)]

    for process in processes:
        process.start()

    for process in processes:
        process.join()

    partial_results = [queue.get() for queue in queues]

    # Reduce
    result = _combine(partial_results)
    return result

def pooled(n, p):
    """
    Partition the range into n parts and run on a pool of p processes.
    """
    # Partition
    parts = _partition(full_range, n)

    # Map
    pool = Pool(p)
    partial_results = pool.map(square_all, parts)

    # Reduce
    result = _combine(partial_results)
    return result

# ---------------------------------------------------------------------------
# A few utilities for partitioning and combining.

def _partition(l, n):
    """
    Partition the list l into n parts, and return a list of the parts.
    """
    count = len(l)
    return [l[floor(i / n * count):floor((i + 1) / n * count)]
            for i in range(n)]

def _combine(partial_results):
    """
    Combine the list of partial results into a final result.
    """
    result = sum(partial_results)
    return result

def _time(f, reps=10, args=tuple(), kwargs=dict()):
    for _ in range(reps):
        t = time.time()
        f(*args, **kwargs)
        print (' ', time.time() - t)

# ---------------------------------------------------------------------------
# Here's the script to generate the results.

if __name__ == '__main__':
    import time

    print('Time for single process:')
    _time(single)

    for n in range(2,9):
        print('Time for {} process:'.format(n))
        _time(multi, args=(n,))

    for p in (2,3,4):
        for n in (p,2*p):
            print('Time for {} partitions over {} process:'.format(n, p))
            _time(pooled, args=(n, p))