Blogs

CIVX: CIVX - Qalthos: Recent Projects: Knowledge

Wed, 08 Aug 2012 03:39:00 +0000

One of the cool features of CIVX that really caught my interest when I first started working on it was the Knowledge DB. Using the magic of Python, Knowledge is a vertical, polymorphic database for storing knowledge about things, which means very little to anyone, even me.

What Knowledge does is it allows you to store anything (or anything Python can pickle, at least), as arbitrary collections of Entities and Facts.

OVC: Open Video Chat - trose: Catching up

noreply@blogger.com (T_Rose) — Thu, 03 May 2012 18:40:00 +0000

Wow, I haven't posted in a long time. Shortly after my last post I had the flu for a week, then I had a job opportunity come up and I spent half a week flying around for interviews. For two weeks after that I was writing docs and doing more research for OVC and I forgot to blog...

So I have some docs up for OVC now explaining how the code works (. I also added to the readme and made additional comments in the code. Hopefully my additions will help the next team that can pick up OVC.

This week and next I hope to add some unit tests to OVC. I'd like to be able to test to make sure the gstreamer interface is working correctly and that the network connection is working. First thing I need to do is figure out how to launch a sugar activity through a unit testing environment. From there it should be easy enough to mock another activity and have it send fake connections.

Update:

I had a conversation with Cerlyn from OLPC about unit testing. It seems like this will be more difficult than I thought...

<trose> can anyone point me to a good reference or example for unit testing sugar activities? specifically testing video and network interfaces?
<Cerlyn> trose: As far as Sugar goes, there is no activity specific testing framework I am aware of; I've been begging for one to be made.
<Cerlyn> Sascha Silbe has been working on unit tests for the core Sugar environment on occasion, but as far as I know progress on that front has been going slow
<trose> I'd like for just some way to start up an activity and then send it fake signals like another activity is trying to connect. Then I could make sure the activity responds correctly
<Cerlyn> many tests Silbe's been working on require user involvement to report pass/fail as well :/
<trose> Cerlyn, well that is frustrating...
<trose> how exactly does sugar start up an activity? is there a way to emulate that?
<trose> or is there a way to say like sugar.startActivity() and get a pointer back?
<Cerlyn> You can start an activity within sugar by running "sugar-launch {fully qualified name of activity here; like org.person.trose.myActivity}"
<trose> Cerlyn, okay
<Cerlyn> That probably wraps a DBUS interface to do it
<trose> but that's cmd right? can i get a handle to inspect the activity?
<Cerlyn> Historically having one activity directly launch another is something that hasn't been done; I do not recall the reasons offhand (not a developer; rather QA)
<trose> Cerlyn, hmm
<trose> i could see how that would cause problems
<trose> this would be a training script though
<trose> *testing
<Cerlyn> trose: In theory, being GTK3 based, GTK-supporting test frameworks should work
<Cerlyn> but graphical testing is obviously not unit level

So there isn't a good way to start up a sugar activity and maintain a handle on it. I may be able to hack around this somehow though. Stay tuned...

OVC: Open Video Chat - trose: Measure Twice Cut Once

noreply@blogger.com (T_Rose) — Thu, 29 Mar 2012 23:17:00 +0000

More research today. I've been mucking around with some of Collabora's code. They are releasing Farstream on Fedora 17 soon. I also found out that there will be a version of Sugar on F17 released in July. Farstream comes with python bindings by default and supports a lot of useful things by default including, RTP and ICE. ICE will be useful as it allows the program to navigate around firewalls and NATs. This should be a valid approach so long as I can get a hold of the development build of Sugar. I don't think I'll have any problems with that though.

If that isn't possible I should be able to continue with current Sugar builds by using telepathy-farsight. This should be very similar to farstream except I'll need to manually pull in the ICE library and perhaps a few other things.

Overall I feel like I'm getting a firm grasp of what needs to happen to get everything working properly. Next week I'm going to start hacking on the OVC code to figure out how I can integrate these new libraries.

OVC: Open Video Chat - trose: OVC Research

noreply@blogger.com (T_Rose) — Wed, 28 Mar 2012 01:37:00 +0000

I dove into the pyGst docs today to try and wrap my head around what exactly is going on. I found the docs to be really confusing at first. Luckily there was an accompanying tutorial that explained things very nicely.

Unfortunately, the tutorial only outlined how to do local streaming. I searched around a little more and discovered that OLPC has a page about gstreamer on the XO's specifically. This page has a lot of information on it. According to that, I need to use a udpsink with an IP to send video to another XO.

At this point I started picking through the code to figure out where the problems might be. I know that somewhere along the way the connection isn't being set up correctly. Either the XO's are failing to get each other's IP's or the video connection isn't establishing.

The part of the code in tube_speak.py that deals with IP's is as such:

@signal(dbus_interface=IFACE, signature='s')
def announce_ip(self, ip):
self.ip = ip

def announce_ip_cb(self, ip, sender=None):
if sender != self.tube.get_unique_name():
self.cb('ip', ip)

This is most likely where the problem is occurring. Basically, the XO's are just spitting out their IP and hoping the other one gets it. If this is the root of the problem, I should be able to solve it with a home spun TCP-like interface where it listens for a return signal.

The other part of the code I've been looking at is here in gst_stack.py:

        # Add udpsink
      udp_sink = gst.element_factory_make("udpsink")
      udp_sink.set_property("host", ip)
        self._out_pipeline.add(udp_sink)
      video_enc.link(udp_sink)

It may be the case that I need to find an alternative to udpsink. Maybe there is something like rtpsink? I'm not sure at this point. Next I think I will start picking at the code to try and determine exactly where it's failing to make the connection.

OVC: Open Video Chat - trose: Migrating OVC

noreply@blogger.com (T_Rose) — Mon, 26 Mar 2012 01:41:00 +0000

Well I haven't talked about Open Video Chat lately. The project has been pretty stagnant for a while. I'm picking it up again this quarter as part of an independent study. Expect weekly updates about the project.

The first order of business is to migrate the project to github. Fedora Hosted was nice but it simply doesn't compare to github for community involvement and collaboration tools. The new repo can be found here: https://github.com/OpenVideoChat/Open-Video-chat.

This quarter I would like to get OVC working with RTP. The current system uses UDP steaming 100% of the time. This means that more often than not the connection packets are dropped since the XO's have horrible connection reliability. RTP fixes this by using a hybrid of TCP and UDP. It uses TCP to ensure the connection is made, then UDP for fast media streaming.

After I complete the RTP conversion I want to focus on getting some more documentation and unit tests for OVC. Right now there is absolutely nothing. When I graduate in May there will no longer be any of the original collaborators at RIT. Providing docs and unit tests will make it easier for other students to pick this up when I'm gone.

OVC: Open Video Chat - trose: POSSE Hackfest '11

noreply@blogger.com (T_Rose) — Wed, 10 Aug 2011 20:16:00 +0000

I'm currently hanging out with the Foss Box and some professors from various local schools. We're trying to teach them about all things open source. I've been playing with Open Video Chat again to see if it works on the newest Sugar build. Sugar 0.92.2 is now running on Fedora 14. Hopefully this means it either has the version of Telepathy we need or we can install it. I was able to get OVC running on two XO's and connected them. When it tried to connect the pipes I got an error though. Seems to be a break at a file open :/ I'll have to look at this more.

Side Note: The XO's automagically start an adhoc network and connect to eachother again! WOOOOO!

CIVX: CIVX - Qalthos: Multiprocessing the PolyScraper

Wed, 29 Jun 2011 22:04:00 +0000

Warning: This post was written at 4AM and contains a technical account of what I have been doing attempting to parallelize CIVX's internals. If you are looking for a more general overview of what I have been doing recently, you are better off looking to another one of my posts. Recently, I have been working on CIVX's PolyScraper, a neat little piece of code designed to be able to read and understand structured text without knowing how the text is structured beforehand. On Thursday, I let a test scrape of a rather large dataset start, figuring it would finish sometime over the weekend and I'd be able to pick it up on Monday. Then Monday rolled around, and the scrape was still running. Worse, it didn't seem to be taking full advantage of the available resources. Running as it was on the boat, it had four cores at its disposal, however it was steadfastly using only one core. Now PolyScraping is not an inherently parallelizable task, but on large datasets it should benefit from some kind of parallelization, particularly when using large numbers of small files (less so with small numbers of large files, those types of tasks are usually sequential as you have a smaller number of resources blocking on read IO). Clearly there were other things to look into, but if I could do this, this would mean a huge win for offline scraping, which was one of the things I had enabled with my addition of local file support to the PolyScraper. This all led me to multiprocessing, a library I'd been wanting to try out in python for some time now. Without going into too much detail, multiprocessing attempts to get around the Global Interpreter Lock by spawning subprocesses instead of threads. The first attempt was written pretty quickly, as I still remember a lot from my Parallel Computing class from a while ago. Indeed the main problem turned out to be SQLAlchemy, or, more specifically our use of sqlite for a backend db. Sqlite is not the most robust of databases and can't really handle multiple processes attempting to write to the db at once. Luke suggested (and I would love to try) moving over to Postgres as we will eventually be doing on the boat, but unfortunately the boat has been 'stuck ashore' for some time now due to an extended outage in CSH's network. In the meantime I have been whittling the process down to what I think is the essentials. In the process I have made a complete mess of the code concerning the PolyScraper, but I should be able to make things at least look like the way they were before too long. At this point, though, I have been working on this project for close to 20 hours today now. Luke is in town and we're all posted up in Remy's new place all hacking on our projects together. With any luck a good night's sleep will clear my head and give me new ideas for tomorrow.

CIVX: CIVX - Qalthos: Back up to Speed

Wed, 22 Jun 2011 02:56:00 +0000

I've almost closed bug #106. I've done all I can do for now, until I can figure out how to attribute actions to senators. Until then, I should comment out the 'actions' tab tomorrow with a note letting whoever tries this next what I've already tried. There is still some work to do in this code, particularly with fixing the hacky Assembly scraping I wrote last year which also broke. However, considering that the Assembly is not currently one of the bits we are trying to expose (and they don't have a nice public API like the Senate), it probably won't get done for a little while. When I left yesterday, I had exposed the senator's social page, but none of the other tabs were showing up. The problem for this turned out to be that the image representing the tab was corrupted and the text beneath it was white, making it overall look like the tab was invisible. Once I got a new image for the tab, they all suddenly appeared, though only the bills had any information. The next thing to fix was the scraping of the committees, whose pages had also changed subtly in the past few months. Fixing this was much less annoying than building them the first time, and I was able to remove a lot of old shims in the code from when I was first being introduced to BeautifulSoup. Some of my later changes made this particularly easy, and since BeautifulSoup is so powerful, I was able to restore access to the data with relative ease. As a bonus, as soon as I had committees back up, the tabs for votes and meetings came with it for free. Suddenly, I was almost there! The final problem I encountered today is actually not a new one, but one we struggled with last year, and one I now feel confident I have actually fixed and understand now. Once I got all the data pulling again, a few of the pages would crash the server with UnicodeDecodeErrors. As a warning, some heavy Python is about to come down UnicodeDecodeError is an error which happens (generally) when attempting to decode a string into a Unicode object. This is generally a great thing to do, as Python strings are generally encoded in the relatively restrictive ASCII, which does not have characters for any of the more exciting characters like accents and non-latin symbols. Unicode has no such restrictions, and indeed has data for many, many more symbols, at the cost of a few more bits of storage per character. So why were we getting this error? The relevant line of code was 's = unicode(s)' and was contained within WebOb code, not something I was going to be able to modify successfully. Still, even this shouldn't be a problem. The purpose of this function is to turn strings to Unicode strings. Except I didn't have a string, I already had a Unicode string. Even this shouldn't be a problem, except that unicode() tries to interpret its input as a string and then turn it into a Unicode string. And while Unicode strings can be easily represented as normal strings, the default of the unicode() function is to try to interpret those strings as ASCII strings, and I had accents in the strings. These strings were representing the names of the senators, so I had to make sure it came out right. In order to solve this, I had to reversibly represent these names as a sequence of ASCII characters. There are a few ways to replace out-of-bound characters when changing strings into lesser encodings, and I had two useful ones to chose from. The obvious one to chose was to change the characters into XML character entities, however this quickly turned out to be insufficient. While é correctly showed up as é on the page, this string is used to represent the name everywhere, including in the internal URL representing the page. And the ampersand was quickly stripped out as a broken argument to the URL, leading to a page for a nonexistent Senator. Looking through the code, there were three distinct uses for the name string. The first, which had started all this, was as an ASCII key to a dictionary which needed to be authoritative but not necessarily accurate. In other words, I needed it to be the same everywhere, but it didn't necessarily need to be the correct name of the senator. The second was the use on the generated web page, which needed to be as accurate as possible to the Senator's actual name, as it is going to be viewed publicly. The third, and the current stickler was the name in the URL. Again, this had to be authoritative but not necessarily accurate. This one, however, had to also only include web-safe characters, of which &, # and ; do not qualify. I mulled this over for a while, thinking up more and more elaborate schemes for intercepting the names before they reached critical areas, but none of it was terribly good coding practice. After far too much thinking, I realized the obvious answer: have separate internal and external names. The system still relies on the senator's name, which is still a questionable practice given the multiple spellings of names that occasionally pop up, (but mostly because I remember this post, which is something you should always keep in mind when programming around names. The display name, on the other hand, has none of the restrictions on characters (though it still needs to be ASCII to display properly), but by using XML entities, we can make any character we want without problems. This was a long path to take to get back to where we were, but I think that I really understand Python's Unicode in a way I never grasped before. This should definitely help in the future as Unicode is a very important part of coding portable applications and that's something I want to do.

CIVX: CIVX - Qalthos: Shelves and Shoves

Wed, 22 Jun 2011 02:56:00 +0000

Today is the day to hit bug #105, another of the leftover bugs from last year. The story goes something like this: there's a lot of data on nysenate.gov that is nice to have, but asking for that info on every call is a little cumbersome. We want to cache as much of the data as we can, particularly the stuff that's not going to change in the next week or longer. Previously, I had implemented a simple pylons cache which was fast, but had no persistent storage, so every time the server went down it pulled all the info again. And due to the way the scrape was written, it pulled all the info for all the senators at once, creating quite a bit of lag before the first page showed up. This clearly wasn't going to be something we could continue to develop with. Now, I know nothing about caching data, so I did some poking around in CIVX to see how it is done elsewhere. Most of the other caches I found in CIVX code were related to caching text feeds, which were not as explanatory as I was hoping. Once I felt I had a handle on how things were done there, I began to try to implement some of it, only to be shown this post on shelve, a Python library for storing arbitrary data. Combined with the decorator, this seemed to do exactly what I wanted, namely provide a permanent storage area for a bunch of data with a configurable expire time. I dumped the code into the dashboard, hooked the proper inputs up and let it run. The results were... promising, but not astonishing. The file storage worked, once the data was cached, we stopped looking to nysenate.gov for data and instead used our own data, even after server restarts. The problem was that the file storage seemed to be slower than the previous memory cache. This is all perfectly reasonable, since disk access is much slower than memory, and a lot of data has to get pulled for each senator. The first obvious thing I could do is to re-enable the memory cache, but this did not seem to help as much as I wanted it to. At this point, Luke popped up in chat to sat that Moksha had a Shove cache it uses for feeds. Sure enough, back in the files I had been poking through earlier, there were some references to Shove. Back to the net, I started to explore what Shove was and how it could help me. It turns out Shove is mostly drop in compatible with shelve, and aims to be a more extensible replacement for it. Once I got a handle on how Shove works differently from shelve (answer, not very), I made a few tiny tweaks and got a version successfully working with Shove and a sqlite backend. This didn't make the end result any faster (well maybe a little, but not much), but there is a lot of room for improvement, particularly if I can hook into Moksha's own stores. Further, Shove has its own abilities to cache items in memory in addition to storing them, which I would like to look into. The best route for efficiencies, I think is to change how the data gets stored in the cache. Currently all the data gets pulled at once, which was done to pacify the pylons cache. However, if I can get individual caches for each senator, then I can pull smaller volumes of data at a time, hopefully speeding up the process. We'll see where I get tomorrow, but so far I'm feeling pretty good about all this.

CIVX: CIVX - Qalthos: Summer of CIVX

Wed, 22 Jun 2011 02:56:00 +0000

So I'm back at RIT for the summer working on CIVX again. There's been a lot of development on Moksha, the software stack CIVX runs on, and not all of it was a trivial update. Still, many of the problems I encountered were definitely my fault, not the least of which was forgetting Arch Linux has switched to Python 3. Once I was up, I looked into fixing the people dashboard we worked on last summer. In the meantime, the NYS Senate updated their API for getting open government data, so I had to figure out the new scheme and try to make it work. Complicating this is that they still don't have easy access to grabbing all the current senators, and the page we look at to get this information changed enough to stop the script from completing. I'm actually surprised how quickly I was able to work this out, though of course this is not even remotely new to me. But despite nearly a year of inactivity on the project I seem to have gotten back into the swing of it pretty well.

CIVX: CIVX - Qalthos: The Last Week

Wed, 22 Jun 2011 02:56:00 +0000

This week has been a mess for a lot of reasons. Let's see what I managed to get through so far. One of the things I got to look at this week is threebean's mokshactl branch of CIVX and Moksha. This is a project to simplify the administration of a Moksha installation. The project grew out of an attempt to easily package Moksha and grew into a much larger system capable of managing most aspects of CIVX at once, and in a very pretty package, too. It has a few problems, but for the most part, it performs quite well, and best of all, it will even integrate itself with Moksha, controlling the necessary aspects of Moksha as well. In less exciting news, I poked around in the people dashboard while I had some time and got Assembly members working again. It didn't take much as I had expected, but served to keep me on task while Remy was in one of the never ending series of meetings he's had this week. On a lighter side of things, I put some fabulous in the CIVX shell today as I was working in it. The mokshactl branch is already fabuloused, and once you see that, there's no coming back. fabulous makes things very pretty with only a little work. Other than that not a lot has gone down. Some work has gone into the polyscraper, but that's nothing worth mentioning at this point. Between that and some internal matters and hours of meetings and my car developing a leak in it's brake line, that's all that went down this week. Tomorrow I get to drive home and hopefully fix my car properly.

CIVX: CIVX - Qalthos: Grokking the Core

Wed, 22 Jun 2011 02:55:00 +0000

This week begins the real dive into the core of what makes CIVX. Today (and yesterday, though yesterday hardly counts as a real day) were spent adding major functionality to the polyscraper, something that's been overdue for a long time now. But what is this magical polyscraper? Well, in short, it's magic. A lot of magic, actually, and that's half the problem. You see, in ye olden days of CIVX, each data source had to have its own scraper, and these were called whenever CIVX decided its data was old enough to get shoved out and replaced with new data. This was all well and fine, except that it took a very long time to get a scraper written for a new data source. You would have to define all the columns, give it a location to look, make sure you understood the site's particular dialect and scrubbed out any irregularities in their data. What the poly scraper does is it replaces all of those individual scrapers and replaces them with one big scraper which is smart enough to deal with any URL it finds. What I've been doing is adding new sources of data to the polyscraper. In particular, yesterday was spent adding the ability to read files off of a local disk and properly store them. This, in turn, exposed a few holes in the underlying framework which needed to be patched. However, this is a vitally important function, as things like the SunlightNY scraper I wrote last year works outside of CIVX proper (in Java, no less) and cannot be thrown into the polyscraper as easily. But I can download the files locally, and then work on them when it is convenient. With proper message passing, I can even seamlessly tell the polyscraper to pick up the files as soon as they are downloaded. Previously to this I had been working at the periphery of CIVX, adding functionality to widgets and individual scrapers. This is my first real push into the core functionality of CIVX, and it is good to see that I really have picked enough up in all this time to really start to understand the underlying structure of everything. Every day I learn more about what goes on inside this machine, and every day marks another set of tools I've learned to wield. I can't wait to see how far I get by the end of the summer.

OVC: Open Video Chat - Jlew: Rochester Great Awards

noreply@blogger.com (Justin Lewis) — Fri, 04 Feb 2011 16:47:00 +0000

Last week I posted that OVC was nominated for the Digital Rochester GREAT Awards.

Fran, Taylor, and I won the Student Achievement Award for our work with OVC.

Taylor and I with our awards on display in the foss box.

From Great Awards

Taylor Rose, Fran Rogers, and Justin Lewis

CIVX: CIVX - Qalthos: Boston

Sun, 23 Jan 2011 12:14:00 +0000

So here we are in OLPC HQ, right in the middle of MIT. It's pretty sweet having Luke around again to hack CIVX with us. I've had a lot to do in the past few days. Remy's been showing me scrapers and models, and I've been helping transition Kate's people dashboard into an integrated component or integrating Rebecca's theme changes on the side. It's been hectic and fun and tough, but I finally feel like I'm contributing to a project, something with substance and goals, not just writing code to accomplish a task like some of my previous co-ops. Being in a team this large helps, especially when we pull in outside help like Luke, but I think it's mainly Remy's infectious excitement for the project. When he gets down to work, one can't help but feel his vision and be excited for the possibilities. Unfortunately, that means I had precious little time to pay attention to the other teams. Three separate groups hacking away at their own projects, tossing ideas about and getting input from a few members upstream, not to mention the whole OLPC offices around the corner- this was a right proper hackathon, and something that makes me excited for the future of these projects.

OVC: Open Video Chat - Jlew: OVC Nominated for the Digital Rochester Great Awards

noreply@blogger.com (Justin Lewis) — Thu, 20 Jan 2011 06:17:00 +0000

Recently, the Open Video Chat team was nominated for the Digital Rochester Great Awards.

The OVC team and one other student from RIT were nominated for the Student Achievement award.

Awarded to a student, currently enrolled in a Greater Rochester higher education institution, who showed leadership and skill in applying or advancing innovative technology for the betterment of themselves, their educational institution, and/or the greater Rochester Community.

Taylor Rose and Justin Lewis
Image by Kenneth Pamatat.

It appears that we were nominated by someone who had seen us showing off Open Video Chat at RIT's Imagine Festival last year where we had a booth set up in the center of the Innovation Center. I personally never herd of this award group and was not expecting to be nominated by a random spectator, I really appreciate it.

Since the nomination, I have been interviewed by RIT University News and just today we made it onto the Homepage of RIT's Computer Science Department.

MAFH: MAFH - Jlew: HFOSS Class Final Presentations

noreply@blogger.com (Justin Lewis) — Fri, 12 Nov 2010 03:09:00 +0000

Tonight was the final presentations for the course. We should be expecting some final documentation updates as well as some activity releases in the new few days!

Lemonade Group in their fancy matching shirts

Project Links

Fortune Hunter: http://wiki.sugarlabs.org/go/Fortune_Hunter
Lemonade Stand: http://wiki.sugarlabs.org/go/Lemonade_Stand
Blocku: http://wiki.sugarlabs.org/go/Blocku
IRC Activity: http://wiki.sugarlabs.org/go/Activities/IRC
Transbot: https://fedorahosted.org/transbot/
PacMath: http://wiki.sugarlabs.org/go/PacMath

Presentation Notes

MAFH: MAFH - Jlew: Fortune Engine and Fortune Hunter

noreply@blogger.com (Justin Lewis) — Fri, 29 Oct 2010 02:08:00 +0000

This post is a bit out of date, but I Finally got the FortuneEngine it's own repository last weekend. The Fortune Hunter repository has grown a lot over the summer due to the vast amount of art and art testing. I figured it was not very friendly to have the engine coupled with that repository if someone wanted to use the engine. It now has its own repository located on fedora hosted. It will still use the same track as fortune hunter.

Now the Fortune Engine has been included in the Fortune Hunter repository as a submodule to keep the engine in the game repository.

I have also made a tarball 1.0 release of the engine as well and tagged it in the repository. My next steps are to upload a copy to pypi and then look into what features I should work on next.

MAFH: MAFH - Jlew: Class Project List

noreply@blogger.com (Justin Lewis) — Thu, 30 Sep 2010 16:54:00 +0000

I have updated our project page on the sugarlabs wiki as well as compiled a list of the class and their blogs, projects, and wiki user pages.

Atkinson, Somara
- Blog
- Project: Fortune Hunter
Case, Nathaniel
- Blog
- Project: Lemonade Stand
Cole, Travis
- Blog
- Project:
Critelli, Matthew
- Blog
- Project: Blocku
Honeycutt, Abigail
- Blog
- Project:
Ito, Kai
- Blog
- Project: Blocku
Kitson, Michael
- Blog
- Project:
Lewis, Justin
- Blog
- Projects: General Support/Helping All Teams
Mengel, Scott
- Blog
- Lemonade Stand
Meschino, Jonathan
- Blog
- Project: Fortune Hunter
Petersen, Lorin
- Blog
- Project Blocku
Rogers, Francis
- Blog
- Project: Irc activity
Rose, Taylor
- Blog
- Project: Translator bot
Thill, Mark
- Blog
- Project: Translator bot
Wagner, Sarah
- Blog
- ProjectLemonade Stand

CIVX: CIVX - Qalthos: Almost There...

Thu, 09 Sep 2010 22:11:00 +0000

Today I worked my way through more of Rebecca's changes to the people dashboard. It was actually in better condition than I had thought last night. The first tab was the only one with the majority of the changes, and most of the changes were easily applied once I understood them- which is what I spent hours yesterday trying to do.

I have a problem with writing quantities of HTML because they end up as giant messes. Even XHTML strict isn't enough for me, though it does come closer. Invariably, any file of reasonable complexity is going to have whitespace inconsistencies, mislaid elements, or even entire sections forgotten. This morning I took a fresh look at the generated HTML and dashboard.mak and tried simply to understand their structure. After really getting into the template version, I began to see what was really necessary and not from Rebecca's blinged up copy. There are still a few bits missing from the final version, but I think I can get them nailed down before I leave today.

CIVX: CIVX - Qalthos: CIVX Stuff

Thu, 09 Sep 2010 22:11:00 +0000

So I've been spending some time finally getting some time to familiarize myself with CIVX and all the parts that make up the system. I'm almost comfortable with how the whole thing works together, though I'm not entirely clear on how some things work to finally get to the screen. Still, I got things running, both the current CIVX build and Kate's widget, though getting the two together is harder than it seems like it should be. On the other hand, I tweaked the GettingStarted page of the CIVX wiki to smooth out a few bumps I ran into during the install process. The Ubuntu instructions are still broken, but I plan going to look into that in the future. In terms of actually getting things done, I haven't done much of substance. I've been immersing myself in CIVX, git, pep8, and all the different things that go into the FOSS BOX. While I've been in the open source community for a while now (and my very first project is almost 1.5 years old), I've never really tried to insert myself into a project that was running in full swing with established work. Somehow this seems different than the places I've worked previously... though that could simply be the distributed nature of the work. With Luke no longer down the hall somewhere, getting information has turned into a more interesting experience when the necessary parts are in his head. I'm looking forward to an exciting next few weeks here, and a fruitful next few years with the information learned from this experience.

CIVX: CIVX - Qalthos: Diving Deep

Thu, 09 Sep 2010 22:11:00 +0000

Yesterday had been spent working on a proper dev branch to CIVX. Today we gave it a home. Today started off trying to get the current dev CIVX running on our server. Now, I had some experience setting up a CIVX instance from getting one running on my box, but my box is mine and I know what's on it. This server is not, so we had a few troubles along the way. Once again the instructions proved insufficient (though not actually bad per se). The server was running Python 2.4 by default, and one of the scripts wanted to be run from Python 2.6, as it was pulling a few things from future. Remy and I spent more than a little while wondering why things weren't happening properly when we had neglected to run a command or two. Luke set us straight at every turn, and I can now say that I actually understand what most of those commands do with respect to the rest of the system. The second half of the day was a bit more interesting. Once CIVX was running, my attention was turned to the scrapers. Most of the scrapers are v2 scrapers, and one of our tasks for the summer is to get things running up to v3. On the first try getting the stimulus watcher scraper running, we had a few problems remembering where everything went and where to call what when. On the second shot getting a scraper for howdtheyvote.ca data, it went much faster. I need to remember to source tg2env when I'm on the server, but mostly, things were good today. Tomorrow we put together the pieces of v3, and keep on moving towards our goals.

CIVX: CIVX - Qalthos: Here be Dragons

Thu, 09 Sep 2010 22:11:00 +0000

Today was the day we got Rebecca on to a git repo towards implementing her changes to the people dashboard. It's only a halfway step, as she isn't entirely using the proper version of the widgets but a hacked one that works locally, and needs to be morphed into something that will work in the context of the CIVX stack. Hopefully soon she will be up to speed and be able to make a more direct submission in the near future. As it stands, this is good stuff, and the end result is definitely usable, even if it's not in the format we want it just yet. Translating one to the other has proven tiresome as well; I spent the better part of three hours just to clean up the first of our tabs. There are a few items I think might not be translatable, things that mainly live in Moksha, but we'll get to that when we get to that. Right now, I need a graphical diff viewer that cares neither for whitespace nor line breaks, just for what's in the code. Good old meld, which I had been using, has proven itself inadequate to the massive differences between these files. Hopefully tomorrow I will have a fresh start from which to sculpt this mess I've constructed into its completed form.

CIVX: CIVX - Qalthos: Introducing: GeoPirate!

Thu, 09 Sep 2010 22:11:00 +0000

So I've received my apparently mandated CIVX nickname, despite it possibly being inaccurate in a week or two. I, along with Pixel Ninja and Python Princess, have been hacking away at our various tasks together for over a week now, and despite the seemingly constant moving from place to place, it's a pretty sweet gig. My work yesterday was some final polishing of the getall script, and my first real push to the geowebdns repository. I'm also working on a supplemental, informational guide to maintenance. Unfortunately, my changes can't quite replace the old script yet, because the import script (the one that actually starts to bring the files into the database and do something with them) is hardcoded to the current files and that script is a bit harder to hack than the download script. The real work was on the documentation, trying to get people to understand my reasoning behind the changes and allow for them to continue my work without too much difficulty.

CIVX: CIVX - Qalthos: Just another ten-hour day

Thu, 09 Sep 2010 22:11:00 +0000

Today was another exciting day in CIVX-land. tl;dr: I spend the day helping other people and being awesome It started out with the latest in a series of attempts at getting Rebecca to a working CIVX repo. As I work through this with her, I am slowly working out how all this works together (though admittedly it is largely me flailing while Rebecca watches). With Luke's help, I eventually realized that the only thing really needed for CIVX to run is python-virtualenv (it can run without, but that involves actually installing python packages to /usr/local, something generally undesired for a development environment). After some question of whether Rebecca had sudo permission, we eventually discovered she did, and a short sudo easy_install virtualenv later, we were ready to start installing the CIVX stack. If you haven't read the CIVX developer's guide (and I think it's probably safe to assume you haven't), it's a bit of a mess. Not actually bad, but short and disorganized. This isn't too bad when you've got a small, fairly tight development group with the main brain usually a ping away on IRC, but as people are finding CIVX, I have taken it upon myself to document every bump in my path. When I was first thrust into CIVX, the page was much sparser with less detail and fewer sections, I have added areas whenever there was a question of how to do what that eventually came down to 'ask Luke'. Any time an arcane set of commands came up I tried to get them on the page with as much information as I could figure out, hopefully someone will take pity on my notes and make them more descriptive. Around this time Kate also had a few questions for me, most of which I could figure out. However, I was still trying to get Rebecca running and hadn't even touched my own computer more than to turn on IRC and look a few thins up for Rebecca. Kate was having some trouble as she was trying to scrape information off the NYS Senate page of senators, neither of which I had done before. Rebecca had other things to do, and getting her Mac up to speed was a lot of wait and pray so I switched over to Kate's task. Now Kate also has a Mac, but she was set up with CIVX long ago and could never quite explain to me how, thanks again to arcane commands. Anyway, her task involved a particular Unicode character in a senator's name not playing well with her scraped name-to-URL converter magic. Having only last night read Falsehoods Programmers Believe About Names off StumbleUpon, I immediately recognized this way as a dead end. Sure, you could force all the current names into this pattern, but it would never last. Some day, a senator would show up such that the senate's conversion script and ours didn't match, and then that senator would disappear from CIVX. After a bit of poking, however, I found that each senator had a link to their contact page right in the div we were scraping. A bit of poking around later, and I had a 100% reliable link to each senator's page, as verified by the senate themselves. Suddenly every senator's page worked, without any of this needless mucking about in Unicode transformations. Which brought us to our second problem. Most (with one important exception) senators have a page hosted on http://www.nysenate.gov, and most have a contact page at /senators/first-m-last/contact. What that page contains, however, seems largely up to each senator. Kate had (and was quite proud of) her 4-line, incredibly complex and unmaintainable regular expression which she used to mangle each page into a regular form. However, as we poked further and further, we found more and more inconsistencies and exceptions to the regular expression. Clearly this was completely the wrong way again, but what was the right way. I suddenly saw an interesting anomaly. Most senators had the contact info were styled exactly the same, despite having quite varying styles otherwise. Kate had already seen that most of the addresses are together in some sort of paragraph tag, and was trying to regexp on the contents of each paragraph on the page. What she hadn't noticed was that every page had a <div class="field-content"> that contained all the contact info. Now that, that was something a bit more to go on. Furthermore, this contained all the contact info- occasionally more, but always the minimum was their District Office and their Albany Office. Furthermore, it was already in some form of HTML, which Kate had previously been stripping and rebuilding manually. If we simply took this HTML as-is and plugged it into CIVX's contact page, instantly every senator had exactly what we (and they) wanted! Well, almost. It was about at this point that Rebecca went home for the day, CIVX not yet working. a few important packages were missing from pypi, keeping us from completing the CIVX setup step so she could get cracking on real CIVX, without needing me to merge every change she wanted to push. Still this left me with more time to work on the regular expressions. Now, most senators worked flawlessly, with two obvious exceptions. the first, and the one I didn't want to tackle just yet, was the senator I mentioned briefly above, the page of Sen. Kemp Hannon. Notice anything different about his page? Well, for one, it's not hosted on nysenate.gov, and for another, he has no explicit contact page. The first made our scraper entirely useless without coding in an exception for senators with separate websites, and the second made such an exception next to impossible to make general, without reverting back to the 'check each paragraph for addresses' method. So Kemp was put on the backburner for now. The other one, which failed somewhat more spectacularly, didn't even break. Rather, the contact page of Sen. John J. Flanagan. Putting aside for the moment the excess content in the div, including the NYS seal, and a few lines about contact information, this is the worst example against automatically generated HTML I have had the misfortune of needing to scrape. Problem 1: <p > <p /><br /> I kid you not, this is on the page a minimum of 20 times in a row so that his Albany Office is so far below the fold so as to be nonexistent. Sometimes there's inline styles, sometimes not. One line has a simple space character instead of the edgier, hipper  . I wanted them all gone.this resulted in five separate regexps so python wouldn't get too greedy and remove all the content. One to replace   with ' ', another to remove all whitespace between a closing angle bracket and an opening one, a third to remove anything matching style="*", a fourth to remove all the (now) empty paragraphs, and a fifth and final one to turn any group of two or more consecutive break tags into a single tag. It is probably fortunate that python would not correctly apply my first attempt which was far less readable, and more of a one-liner, as I don't know if I could have understood it now had I not broken it into its component parts. Problem 2 is a bit more of a WTF moment, both beautiful and frightening, so I will reproduce it here verbatim:

<P style="TEXT-ALIGN: center"><SPAN style="COLOR: #012849; FONT-SIZE:
18pt"><SPAN><SPAN><SPAN style="LINE-HEIGHT: 115%; FONT-FAMILY:
'Calibri', 'sans-serif'; COLOR: #012849; FONT-SIZE: 16pt;
mso-fareast-font-family: Calibri; mso-ascii-theme-font: minor-latin;
mso-fareast-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;
mso-bidi-font-family: 'Times New Roman'; mso-bidi-theme-font:
minor-bidi; mso-ansi-language: EN-US; mso-fareast-language: EN-US;
mso-bidi-language: AR-SA"><SPAN><STRONG><SPAN style="FONT-FAMILY: Times
New Roman">District Office<BR /></span></strong></span><SPAN
style="COLOR: #012849; FONT-SIZE: 16pt"><SPAN style="FONT-FAMILY: Times
New Roman">260 Middle Country Road, Suite 203<BR />Smithtown, New York
11787<BR />631-361-2154<BR />631-361-5367
FAX</span></span></span></span></span>

For those of you following along at home, that's P(text-align) > SPAN(color, font-size) > SPAN > SPAN > SPAN(line-height, font-family, color, font-size, a bunch of other font styles) > SPAN > STRONG > SPAN(font-family) > SPAN(color, font-size) > SPAN(font-family) Naturally, my first order of business was to remove every single span from the HTML we take in. Because, frankly, this is preposterous. We already (by problem 1) strip out all the style information, because frankly, we don't need it, so this mess just turns into six nested spans, not a very useful thing. Suddenly, the HTML coming out of the sanitizer is much more compact, and not just because of all the breaks and paragraphs I took out. By the time I finished with this, it was about an hour after most everyone else had left. I spent the next half hour checking that my sanitizer didn't break existing pages (it did, but only minorly) and making sure my code was legible. At that point, almost ten hours after I had started, I sat back, committed my final changes, and decompressed. *This*- this is why I love open source.

CIVX: CIVX - Qalthos: More Conference Calls

Thu, 09 Sep 2010 22:11:00 +0000

Every time I go through one of these calls I feel like I come out knowing less about the topic discussed. The topics aren't hard, and I can now follow along and understand most of what they're talking about, but most of the technologies are things I haven't really looked at before. It's interesting, and it's exciting, but it's a little wearying too. The majority of the talks seem to revolve around the use of Amazon EC2 instances, of which I know nothing. On my own tasks, the semifinal versions of the getall script and the associated documentation got pushed to geowebdns' repository today. I feel pretty good about the changes I made, even if it doesn't prove useful to the current task, I have opened up some small part of someone else's open source program, and that's really cool.

CIVX: CIVX - Qalthos: Hello, world...

Thu, 09 Sep 2010 22:10:00 +0000

This is my first posting despite having been here for a week, because the zaniness that was my first week did not allow for such silly things as introductions. Now I finally have some of the infrastructure things worked out, and can report on my findings. Work has been progressing on a scraper for shapefiles for the Senate. I've been fairly successful in getting a large amount of data, (up to 7.6 GB of zip files!) so my task today was to clean up the script and make it readable to commit it back upstream. I had started by adding more loops to the script for special cases, but that meant repeating the core download code. Realizing this, I made a generalized downloader function which gets called by each loop, simplifying individual loops into a simple

'for URL in $URL_LIST; download $URL $FILE; done'

Well, I think it's simpler... Each time I tweak away at this file, my bash skill gets better and better, but whether any of this is going to stick around is another question. I always forget the simple bashisms, like for loops and conditionals, and I can never remember quite how variables work. But as I work more with this file, I get more comfortable with these conventions. Though I still occasionally pine for the occasional pythonic statement (and perhaps also for the fjords).

MAFH: MAFH - Jlew: Busy Days

noreply@blogger.com (Justin Lewis) — Mon, 09 Aug 2010 21:50:00 +0000

We are approaching the end of this summer block, trying to wrap things up. I have been jumping around a few projects lately, Fortune Hunter, my School Server, and a csi website.

Local Hackfest

We had a hackfest in Center for Student Innovation on the 4th of August. Buffalo Chicken Pizza, groove bot blasting music, and lots of coders and a few projects; It was a great time.

Fortune hunter team spent the day adding some touches to the animation engine. By the end of the hackfest I was able to merge the art team's branch into the main. Having 415 edits, only one merge conflict is quite impressive. Some minor engine changes followed in the next couple of days to return backwards compatibility with Lemonade (which is now being updated to the new animation system).

Taylor and Eric were working on the CSI web app adding the ability to link up nodes as well as a few other nice tweaks. Jon Schull and I were also helping out in the early hours of the hackfest, planning some of the features as well as going through the code with Taylor and Eric.

Civx had the whole crew plus some like Luke Macken coding away. One of the tasks for Nate was adding javascript scraping.

Upcoming Research Symposium

Well we have all been really busy preparing for our Research Papers, Posters, and Presentations for RIT's Undergraduate Research Symposium that is being held on August 13th.

The two fortune hunter groups have decided to have one poster and one presentation that will each cover both topics.

Infrastructure

We have been doing a lot of things on our infrastructure side. We got our three sugarlabs servers up and they just need some configuration and we will be hosting a sugar activities mirror.

I have also been busy working on our own hosted system. We have our own trac and git repositories hosed on schoolserver. We even have a club here at RIT testing out the system. So hopefully this will be the start of a very valuable resource.

We have also given Dave Silverman the task of building a new logo for FOSS @ RIT. Well Dave delivered, so here is our new logo:

Some other things I have been working on is a gitorious config script to allow easy modification of our git repos on schoolserver. My long term goal of this is to write/modify a trac plugin to allow git management from trac.

Gource Visualization

I have also been compiling a gource visualization that has all of our projects in one feed. This required me to write a script that can combine the git logs of a few git repos into one log.

Watch on YouTube

MAFH: MAFH - DaveSilver: Integration...complete? Not quite...

Thu, 05 Aug 2010 16:11:49 +0000

So it's been a while since I posted anything to this blog. I hope you guys haven't missed me too much. This week has been pretty eventful. The last three days Kevin and I have been working to integrate the animation stuff into the Game Engine that he and Jlew made. This was a large task but not a vastly challenging one. It was more about determining the exact process we would use and doing that for everything, than it was about figuring out a unique process for each situation.

MAFH: MAFH - Blitzkev: Hackfest Success

Thu, 05 Aug 2010 15:27:55 +0000

Actions speak louder than words so check below for commits made and a screen cap of some new animations running in the integrated animation/game engine.

http://git.fedorahosted.org/git/?p=fortune_hunter.git;a=summary

http://imgur.com/9f9Ivl.jpg

MAFH: MAFH - Blitzkev: Further success with integration

Mon, 02 Aug 2010 22:50:18 +0000

Today Dave looked into what was causing the randomness in the drawing hierarchy, and traced most of it to a dictionary. We pushed a small fix for that issue and are looking at fixing the other issues that cropped up. On my part I finished porting the battle menu, started the magic menu (which has a lot of work) and helped Dave with the dungeon drawing methods.

MAFH: MAFH - Blitzkev: Integration of drawable objects and fortune hunter

Fri, 30 Jul 2010 20:10:58 +0000

After finishing produce puzzle on Monday (the activity can be found here: http://activities.sugarlabs.org/en-US/sugar/addon/4322) I have been working hard with Dave to get fortune hunter completely changed over to using his drawable object and scene classes instead of the old surface blits. Our 50+ git commits can be found here http://git.fedorahosted.org/git/?p=fortune_hunter.git;a=shortlog;h=refs/... .

MAFH: MAFH - DaveSilver: Every time you fix one problem, another crops up...

Thu, 29 Jul 2010 16:46:59 +0000

As you may be able to guess from my blog title I have been running into more issues while trying to integrate my stuff into Jlew and Blitzkev's engine. Mainly we are having issues getting things to display because the entire engine has not yet been ported over to the new system. This means that some things are using our sprite and sprite group system while other things are using the old blit and flip method. For some reason this is causing the stuff to not display the same way every time we test and thus makes it very hard for us to pin down the exact issues we are having.

MAFH: MAFH - DaveSilver: Integration... sucks...

Tue, 27 Jul 2010 15:34:42 +0000

So today I finally finished all of the classes and functions of the animation system. My next task is to integrate all of the classes into the game engine that Jlew and BlitzKev have built. This task is pretty hefty and will probably take a day or so to complete. It involves me finding every location where my stuff would be used and then figuring how best to make it work in those locations. It's a big task but I know I am up to it.

MAFH: MAFH - Blitzkev: Mon July 26th - Produce Puzzle done

Mon, 26 Jul 2010 22:00:11 +0000

After a lot of hard work and 13 commits, I've finally finished porting my modified version of produce puzzle to the game engine. I also got the game working as an .xo bundle on the xo. Uploading to activities.sugarlabs as soon as I'm able.

More fortune hunter polishing coming next, as well as finishing our abstract before Sunday night.

MAFH: MAFH - DaveSilver: T-Minus Three Weeks...

Mon, 26 Jul 2010 15:52:09 +0000

So we officially have only three weeks left to finish this project. That seems like a small amount but I think we are pretty much on track for that. Last week we discovered that using the images object type was a bad idea and decided we needed to convert our system to using sprites. Thankfully this does not require too much hard work on our end. I am currently working towards this goal.

MAFH: MAFH - jtmengel: Tales of Thor's Day

Thu, 22 Jul 2010 19:06:13 +0000

Alright, time for the Thor's Day Blog, also know as Thursday by some nations that don't recognize the day's pagan origins. Ahem, but anyways, today was a day of killing off the tests we started yesterday on using classes, inheriting sprite, in conjunction with dirty lists.

MAFH: MAFH - DaveSilver: Thurs time's the charm...

Thu, 22 Jul 2010 18:40:04 +0000

So today we worked on getting our system working so that we would not have to refresh everything on screen every single frame. We were pretty successful.

CIVX: CIVX - kayeight: I made success!

Thu, 22 Jul 2010 13:20:02 +0000

Yesterday I completed the meetings and votes tabs of the person widget, which were a cinch with the list of committees successfully scraped by the masterful Geo Pirate. I can now safely say we're nearly done with this widget, save for the calendar tab (which appears to be on the back burner now) and a few minor ordering issues. Our next target is a poly form widget and a location dashboard, parts of which are already working.

MAFH: MAFH - Jlew: Fortune Hunter Dungeon Advancing

noreply@blogger.com (Justin Lewis) — Wed, 21 Jul 2010 21:45:00 +0000

I finally started implementing the ability to go to the next dungeon (ticket 44).

I didn't close the ticket because there seems to be some bug in the code on level two that gets you stuck in a room on level two.

Otherwise, I have been doing a lot of little things. Hopefully I'll get this figured out soon.

MAFH: MAFH - Blitzkev: Wed July 21st

Wed, 21 Jul 2010 20:52:02 +0000

Today was the weekly storyteller meeting. We laid down plans for the next week - in short:
I hope to finish produce puzzle by the end of the day Friday, and if not then, early next week. After that Jlew and I will go back to polishing fortune hunter as our flagship example of what the fortune engine is capable of. We will also be extending help to the Games for Science group and to the animation team.

Today I worked further on the product puzzle game:
http://git.fedorahosted.org/git?p=fossrit.git;a=commit;h=5b98697a951e046...

MAFH: MAFH - jtmengel: Bloging blogs, everyday

Wed, 21 Jul 2010 19:33:18 +0000

Design was previously thought to be complete and implementation could begin, but wait! There's more!

We had previously moved on from dirty blitting content that we fully understood the subtleties therein, full of bravado and confidence that, as you will soon learn why, quickly evaporated. As our plans were attacked by rabid weasels.

I carefully chose this metaphor to perfectly illustrate where we are, or course.

CIVX: CIVX - kayeight: a different point of view

Wed, 21 Jul 2010 13:15:59 +0000

Yesterday, while becoming increasingly frustrated at the NY senate's website as I tried to scrape committee information, Nate suggested a much easier solution. I had been attempting to visit each senator's page and get their committees individually, but naturally, the website is formatted so inconsistently that no two senators list their committees in the same place or manner. I whiled away a number of hours in this direction before Nate had the idea of searching from the other direction - visiting each committee page and getting its members.

MAFH: MAFH - DaveSilver: Things Keep Getting Complicated...

Wed, 21 Jul 2010 13:15:17 +0000

So yesterday was primarily successful however towards the end of the end we started running into some problems. Basically as we got to the end of the day we found that some of the methods we were using, or planning on using, were not going to be as successful as we thought. You see, we intended to use a "dirty" update method. This means that instead of updating the whole screen every frame, it only updates the areas covered by objects that actually need to be updated. To do this, we would simply create a list of the rectangle objects associated with what needs to be updated.

MAFH: MAFH - DaveSilver: Moving On to Larger Things...

Tue, 20 Jul 2010 18:50:43 +0000

So yesterday we finally implemented rotation correctly and it looks great. Once we had that down we also implemented Rendered Transparency so that users could make more complex looking images, and the convert function so that images can be optimized more easily. After we got that working we moved on to determining how we would implement the animation classes into the larger Game Engine.

MAFH: MAFH - Blitzkev: July 20th

Tue, 20 Jul 2010 16:51:13 +0000

I've been emailing Abbi working out how the Produce Puzzle main game should work and function. I have a good idea now how to move forward and lay it out. I'm working with the animation guys to integrate their Drawable Object classes into fortune hunter; and I'll use their classes for all the images in produce puzzle!

MAFH: MAFH - jtmengel: Incorporating what we've found

Tue, 20 Jul 2010 13:18:11 +0000

In these last two weeks, Dave and I on the front end team have made progress on outlining what we want to implement and, more importantly, what we will leave out to streamline performance; We've been bouncing between two phases of our design process, the first being fervently adding features and functionality and the second being aggressively purging what isn't absolutely necessary.

MAFH: MAFH - Blitzkev: Produce Puzzle Progress

Mon, 19 Jul 2010 18:38:10 +0000

Latest git commit can always be found here: https://fedorahosted.org/fossrit/browser/produce_puzzle

Finished the main menu (plus my custom graphic :D), from which the old code can now be run. I'm cleaning the old code up to remove all pygtk issues and be engine compliant.

CIVX: CIVX - rberent: rberent

Mon, 19 Jul 2010 15:12:38 +0000

Spending a lot of time making the CIVX interface more sleek and cutting-edge by implementing and tweaking some neat jQuery widgets that display content in a dynamic and clear way, making it easy to display content on one page without causing the user to have to scroll. I will add to this post later when I’ve finished the latest widget so you can see it in action. All widgets are re-branded to match the new CIVX theme, but all code we use, including our widget code, is available for any user to view, download, implement, or whatever. Everything we use everywhere on our site is free to the public, hence keeping up with our truly, completely open source mission.

MAFH: MAFH - DaveSilver: Rotation is My Chief Concern...

Mon, 19 Jul 2010 13:03:15 +0000

So a new week has begun and we are still getting rotation to work. We have a few ideas and supposedly JT got pretty far on it last Thursday but, with me not being in the FOSS Box that day, we were not able to truly work together. He has not come in yet but will probably be here soon and thus we will be able to bang out this final piece of the "puzzle". Beyond that we will also be working on integrating our system with Jlew's game engine today. We are scheduled to have a meeting sometime today on how they can work together.

CIVX: CIVX - kayeight: CIVX-ing away

Mon, 19 Jul 2010 12:44:29 +0000

Lately we have been trying to both put a swanky new theme on the CIVX website as well as wrap up the person dashboard. While Rebecca has been doing an amazing job on the former, I have been working on the latter. I ran into a few snags earlier last week when I experienced the results of not pulling from the repository often enough (note to self: don't do that again) and had problems merging my code with the large amount that had been committed since I last pulled.

MAFH: MAFH - Jlew: Fortune Engine profiler and other projects

noreply@blogger.com (Justin Lewis) — Thu, 15 Jul 2010 21:28:00 +0000

Yesterday I resolved the profiling ticket (ticket 31). This adds a basic profiling system to the game that can be run through the game console

This introduced a bug that I fixed today (changeset).

Kevin and I also started working on our paper for the symposium at the end of the semester.

Aside from Fortune Hunter and the Engine, I have also started working on porting Nate's Lemonade Stand game over to the engine.

Python backwards compatibility drives me nuts. The xo 1.0 runs python 2.5 or older where the newer ones use python 2.6. In the game engine I was using the keyword "as" in the exception handler where in python 2.5 the keyword is ",". So I had to make a commit to convert all my code so that it doesn't use the newer "as" keyword (changeset).

Pygame has a similar issue... I had to remove using pygame.Color and just use the tuple format for the rgb color values in our game (changeset 1 and 2)

Other then that, I have also been helping Taylor on a project for the CSI. This project is like making a facebook/social networking site driven around projects and events.

MAFH: MAFH - DaveSilver: Better late than never....

Thu, 15 Jul 2010 19:20:53 +0000

So last night I got food poisoning when I ate some sour fruit and because of that I was not able to physically be at the Foss Box today. Despite that I was still able to get some work done. Basically yesterday I spent the whole day building the Scene class which was the final piece that needed to be completed of the animation system. With the code complete I spent today running tests to make sure things work and to make sure there was nothing I did wrong. Obviously I did do stuff wrong at first but it has now all been fixed and the class is fully functional.

MAFH: MAFH - Blitzkev: Produce Puzzle

Thu, 15 Jul 2010 19:07:37 +0000

The first thing I did today was to fix two bugs in the new MAFH, https://fedorahosted.org/fortune_hunter/ticket/39 and one related to targeting enemies that no longer exist. I spent a few hours helping the games for science team, getting them up-to-date on git and how the game engine works.

I talked to Abbi Honeycutt, a developer of the Math4 game produce puzzle http://wiki.laptop.org/go/Produce_Puzzle, and am now working on porting that over as another proof of concept. This is what I'll be spending the next 6 or so work days on.

MAFH: MAFH - Blitzkev: The end of Fortune Hunter

Wed, 14 Jul 2010 19:17:43 +0000

Ticket #17 has officially been closed (https://fedorahosted.org/fortune_hunter/ticket/17). There will be more discussion on what I can further port over to the Game Engine; as Justin and I believe that porting Fortune Hunter over as a proof of concept has been a success.

MAFH: MAFH - DaveSilver: Animation Systems are...Coming...

Wed, 14 Jul 2010 12:40:10 +0000

So as of last night the animation systems are in a pretty good place. I have most of it done and am moving on to the final piece today. Overall things are going pretty well with it so far. The rotation stuff is still giving us problems but we have decided to put it on hold until we have everything else complete. This way we will not be held up by it and fall behind schedule. I expect that once we get away from the problem and stop thinking so much about it we will come up with a solution because that's just how these things tend to go but who knows.

MAFH: MAFH - Jlew: Lemonade finds bug in Fortune Engine

noreply@blogger.com (Justin Lewis) — Wed, 14 Jul 2010 02:44:00 +0000

Today I spent a lot of time working on the lemonade stand activity. The activity is almost back to its original state but in the game engine. All that is left is the change counting mini-game at the end of each day. After that it is just improving the gui.

While working on lemonade, I found a bug in the game engine. This bug caused the game to appear to freeze when the game console was summoned in non-animation mode (ticket 38). The ticket has not been closed yet due to the fact the problem remains when you close the console until the next event sets the screen dirty.

MAFH: MAFH - Blitzkev: Week 6

Tue, 13 Jul 2010 21:06:12 +0000

Battles are almost 100% finished. Players can now kill all enemies and move further on into the dungeon. The one thing left to implement is item drops. The whole process has been a very good proof of concept for the game engine itself.

MAFH: MAFH - Jlew: Game Engine and Lemonade Stand

noreply@blogger.com (Justin Lewis) — Mon, 12 Jul 2010 22:30:00 +0000

Today I started out by fixing a bug in the game engine that caused the timer to give an inaccurate number of millisecond passed when not in animation mode (Ticket #36). I also added some more commenting to the game engine (changeset).

After that I started recoding lemonade stand so that it will eventually run using the game engine. In its current state ( and the other commit I did today) it is able to run the basic process day loop. This includes buying items (or being forced to buy as many as you can if you try to buy to much making the game more interesting), decaying the inventory, process a weather change, and a random inventory event. Next I need to to implement the simulation portion where people go and buy the items and go into a mini game.

From there I get to start working on the main gui and some of the mini games.

CIVX: CIVX - rberent: rberent

Mon, 12 Jul 2010 16:52:25 +0000

In Boston, we hacked all day with the folks at OLPC (One Laptop per Child), the whole CIVX team (including Luke Macken!) and the rest of our team from RIT. Basically, my task was to make the user dashboards (both people dashboards and state dashboards) look “pretty”. This involves extensive changes to the CSS files, as well as applying and tweaking the custom theme I previously blogged about creating using jQuery’s Theme Roller. Some of the styles that were set up based on the themeroller generated imagery and others generated css code. I was able to go in and tweak some of the settings without having to re-create an entire new theme, which was pretty awesome and saved me lots of time. I also used CSS to re-position a lot of the elements on the dashboards, apply hierarchical typographic styles, text treatment, link behavior, and more.

In addition to hacking, we got a tour of the OLPC Headquarters which was both moving and incredibly interesting to me having come into this job after the development of Justin, Taylor and Fran’s Open Video Chat for the XOs (OLPC laptops). I learned a lot and had a nice time networking and looking around.

On another note, the trip was a great bonding experience for our team here this summer. I think we all had a good time hanging out and spending time with one another outside of the office. Dinners were fun, and several of us went to a drum and bass club where we listened to some good music and got to know each other better. An especially awesome occurance was the surprise meet-up with my college roommate and great friend Keith Simmons, a good friend and former student of SJ’s (Steve Jacobs) and an amazing hacker and pixel ninja himself. I also planned a meet-up with another friend of mine from college, Mark Cuddy, a former Industrial Design RIT grad, and both of them came out and joined us for fun on the last night in Boston. Overall, Boston was a great experience and a lot of work was done with a lot of people collaborating!!

CIVX: CIVX - kayeight: hackfest.success == true

Mon, 12 Jul 2010 13:04:01 +0000

As you may have read, this past week the entire FOSS-box roadtripped to Boston for a hackfest at OLPC headquarters. Though we don't work on the XO laptops, CIVX was also able to tag along, and it's a good thing we did - the opportunity to work in person with Luke again was invaluable. It's awesome what can get done over the course of eight hours in a conference room with a bunch of hacker extraordinaires!